Error when injecting urls

I am new to Sparkler. I begun with the requirements provided and everything worked well. Then, when I started to inject some urls, I got an issue.
Here are the steps that I followed to inject urls:

cd to the root directory of the project
docker run -it sparkler-local
/data/solr/bin/solr
/data/sparkler/bin/sparkler.sh
java -jar sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt

The output that it must appear is:
2016-06-07 19:22:49 INFO  Injector$:70 [main] - Injecting 2 seeds
>>jobId = sparkler-job-1465352569649

But, I got this output:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
I don't understand what caused this problem and how can I resolve it.  I will be very gratefull  if I get an answer. I look forward to your reply.
Thank you.

I've not run the docker container, but basically you're not providing a
path to the jar so it can't find it, either provide the path or I suspect
sparkler.sh does the job for you.
java -jar /data/sparkler/...... I guess is what you're after.
Tom
On Fri, Aug 4, 2017 at 10:58 PM, User12300<EMAIL_ADDRESS>wrote:

I am new to Sparkler. I begun with the requirements provided and
everything worked well. Then, when I started to inject some urls, I got an
issue.
Here are the steps that I followed to inject urls:

cd to the root directory of the project
docker run -it sparkler-local
/data/solr/bin/solr
/data/sparkler/bin/sparkler.sh
java -jar sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt

The output that it must appear is:
2016-06-07 19:22:49 INFO  Injector$:70 [main] - Injecting 2 seeds

jobId = sparkler-job-1465352569649

But, I got this output:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
I don't understand what caused this problem and how can I resolve it. I
will be very gratefull if I get an answer. I look forward to your reply.
Thank you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/USCDataScience/sparkler/issues/124, or mute the
thread
https://github.com/notifications/unsubscribe-auth/AAGUeG0IrrlWm_X2B4tabPpv9UkQ87RBks5sU5QLgaJpZM4OuO08
.

--
Tom Barber
CTO Spicule LTD
<EMAIL_ADDRESS>http://spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316
US: +18448141689
https://leanpub.com/juju-cookbook

Thank you for your reply. I tried to add the path to the jar but it still doesn't work unfortunately.
When I use sparkler.sh it shows this output:
sparkler.sh: command not found
So, I tried this: bash sparkler.sh and the output was:
Sub Commands:
  inject : edu.usc.irds.sparkler.service.Injector
         - Inject (seed) URLS to crawldb
   crawl : edu.usc.irds.sparkler.pipeline.Crawler
         - Run crawl pipeline for several iterations

And when I continue to inject, it shows the same error:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar

java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar
is what you're after. Although it seems to me like solr installed with the
wrong permissions currently when i built it, I'll fix that.
Tom
On Sat, Aug 5, 2017 at 6:09 PM, User12300<EMAIL_ADDRESS>wrote:

Thank you for your reply. I tried to add the path to the jar but it still
doesn't work unfortunately.
When I use sparkler.sh it shows this output:
sparkler.sh: command not found
So, I tried this: bash sparkler.sh and the output was:
Sub Commands:
inject : edu.usc.irds.sparkler.service.Injector
- Inject (seed) URLS to crawldb
crawl : edu.usc.irds.sparkler.pipeline.Crawler
- Run crawl pipeline for several iterations
And when I continue to inject, it shows the same error:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320455398,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGUeJzJxEj4x6AkzFSx-1Z0qiEsYUkxks5sVKHVgaJpZM4OuO08
.

--
Tom Barber
CTO Spicule LTD
<EMAIL_ADDRESS>http://spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316
US: +18448141689
https://leanpub.com/juju-cookbook

As you'll see here it works now:
https://asciinema.org/a/2tslGQY2MGyaGlVy8rEU0fohC
The only change I have to make is cd to /data/sparkler before running the
crawl.
Tom
On Sun, Aug 6, 2017 at 3:32 AM, Tom Barber<EMAIL_ADDRESS>wrote:

java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-
SNAPSHOT.jar
is what you're after. Although it seems to me like solr installed with the
wrong permissions currently when i built it, I'll fix that.
Tom
On Sat, Aug 5, 2017 at 6:09 PM, User12300<EMAIL_ADDRESS>wrote:

Thank you for your reply. I tried to add the path to the jar but it still
doesn't work unfortunately.
When I use sparkler.sh it shows this output:
sparkler.sh: command not found
So, I tried this: bash sparkler.sh and the output was:
Sub Commands:
inject : edu.usc.irds.sparkler.service.Injector
- Inject (seed) URLS to crawldb
crawl : edu.usc.irds.sparkler.pipeline.Crawler
- Run crawl pipeline for several iterations
And when I continue to inject, it shows the same error:
Error: Unable to access jarfile sparkler-app-0.1-SNAPSHOT.jar
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320455398,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGUeJzJxEj4x6AkzFSx-1Z0qiEsYUkxks5sVKHVgaJpZM4OuO08
.

--
Tom Barber
CTO Spicule LTD
<EMAIL_ADDRESS>http://spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316 <+44%2056%200364%201316>
US: +18448141689 <(844)%20814-1689>
https://leanpub.com/juju-cookbook

--
Tom Barber
CTO Spicule LTD
<EMAIL_ADDRESS>http://spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316
US: +18448141689
https://leanpub.com/juju-cookbook

Thank you fo your help. The link was very helpful.
But, I still have some questions since I didn't understand what caused the problem for my case and I didn't find the right outputs yet.
Do I have to start solr after running this command: docker run -it sparkler-local ?
When I run (following the link that you sent) : 1) docker run -it sparkler-local
2) /data/solr/bin/solr start
it shows this error:
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs

But when I do it in this order:  1) cd to the solr directory
2) ./bin/solr start
3) docker run -it sparkler-local
solr started succesfully. However, when I continue injecting, it still doesn't work since when using this command to inject urls: java -jar /data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar inject -sf seed.txt
It shows this error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at edu.usc.irds.sparkler.Main$.main(Main.scala:47)
	at edu.usc.irds.sparkler.Main.main(Main.scala)
Caused by: java.nio.file.NotDirectoryException: seed.txt is not a directory
	at edu.usc.irds.sparkler.service.Injector.stackListFiles(Injector.scala:126)
	at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:86)
	at edu.usc.irds.sparkler.base.CliTool$class.run(CliTool.scala:34)
	at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:44)
	at edu.usc.irds.sparkler.service.Injector$.main(Injector.scala:147)
	at edu.usc.irds.sparkler.service.Injector.main(Injector.scala)
	... 6 more

what I understood from your reply is that you have changed permissions of solr in the project. So, I tried to redownload the project and I run: mvn clean install. This was the output:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war (default-war) on project sparkler-ui: Execution default-war of goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war failed: basedir /home/student/work/sparkler_project/sparkler-master/sparkler-ui/banana/dist does not exist -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException

I am having problems to understand the errors and what should I do to resolve that.
Thank you.

Sorry my fault, git pull and try again please.
Tom
On Mon, Aug 7, 2017 at 3:41 PM, User12300<EMAIL_ADDRESS>wrote:

Thank you fo your help. The link was very helpful.
But, I still have some questions since I didn't understand what caused the
problem for my case and I didn't find the right outputs yet.
Do I have to start solr after running this command: docker run -it
sparkler-local ?
When I run (following the link that you sent) : 1) docker run -it
sparkler-local
2) /data/solr/bin/solr start
it shows this error:
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
at java.nio.file.Files.move(Files.java:1395)
at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs
But when I do it in this order: 1) cd to the solr directory
2) ./bin/solr start
3) docker run -it sparkler-local
solr started succesfully. However, when I continue injecting, it still
doesn't work since when using this command to inject urls:java -jar
/data/sparkler/sparkler-app/target/sparkler-app-0.1-SNAPSHOT.jar inject
-sf seed.txt
It shows this error:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at edu.usc.irds.sparkler.Main$.main(Main.scala:47)
at edu.usc.irds.sparkler.Main.main(Main.scala)
Caused by: java.nio.file.NotDirectoryException: seed.txt is not a directory
at edu.usc.irds.sparkler.service.Injector.stackListFiles(Injector.scala:126)
at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:86)
at edu.usc.irds.sparkler.base.CliTool$class.run(CliTool.scala:34)
at edu.usc.irds.sparkler.service.Injector.run(Injector.scala:44)
at edu.usc.irds.sparkler.service.Injector$.main(Injector.scala:147)
at edu.usc.irds.sparkler.service.Injector.main(Injector.scala)
... 6 more
what I understood from your reply is that you have changed permissions of
solr in the project. So, I tried to redownload the project and I run: mvn
clean install. This was the output:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war (default-war) on project sparkler-ui: Execution default-war of goal org.apache.maven.plugins:maven-war-plugin:2.1.1:war failed: basedir /home/student/work/sparkler_project/sparkler-master/sparkler-ui/banana/dist does not exist -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
I am having problems to understand the errors and what should I do to
resolve that.
Thank you.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/USCDataScience/sparkler/issues/124#issuecomment-320682912,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGUeFjm2XyBfyTdJYiPkgQqHzoh603bks5sVyIogaJpZM4OuO08
.

--
Tom Barber
CTO Spicule LTD
<EMAIL_ADDRESS>http://spicule.co.uk
@spiculeim http://twitter.com/spiculeim
Schedule a meeting with me http://meetme.so/spicule
GB: +44(0)5603641316
US: +18448141689
https://leanpub.com/juju-cookbook

Ok mvn clean install is running successfully. However, the error persists when I run: docker run -it sparkler-local then /data/solr/bin/solr start.
The error:
sparkler@94ac27b70903:/data$ /data/solr/bin/solr start
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs

ERROR: Logs directory /data/solr/server/logs is not writable. Exiting

Can you help me to resolve it

Same error also here:
sparkler@94ac27b70903:/data$ /data/solr/bin/solr start
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr_gc.log.0.current -> /data/solr/server/logs/archived/solr_gc.log.0.current
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.archiveGcLogs(SolrCLI.java:3903)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3872)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old GC logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr-8983-console.log -> /data/solr/server/logs/archived/solr-8983-console.log
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.archiveConsoleLogs(SolrCLI.java:3932)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3875)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed archiving old console logs
Exception in thread "main" java.nio.file.AccessDeniedException: /data/solr/server/logs/solr.log -> /data/solr/server/logs/solr.log.1
	at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
	at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
	at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:457)
	at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
	at java.nio.file.Files.move(Files.java:1395)
	at org.apache.solr.util.SolrCLI$UtilsTool.rotateSolrLogs(SolrCLI.java:3975)
	at org.apache.solr.util.SolrCLI$UtilsTool.runTool(SolrCLI.java:3869)
	at org.apache.solr.util.SolrCLI.main(SolrCLI.java:257)
Failed rotating old solr logs

ERROR: Logs directory /data/solr/server/logs is not writable. Exiting

@thammegowda do you have any idea what's up here?

@chrismattmann
It's permission issue due to user mismatch.
we were using root user earlier (inside docker), and everything was fine.
However,  recently the user name was switched to sparkler (reason: to take away super user privileges from sparkler processes).
That's where these permission issues appeared
@buggtb tried to fix it here by giving permission to sparkler user: https://github.com/USCDataScience/sparkler/commit/fd27abed46a948d095446e3425641f1f7be02a96
However, it is not completely fixed!
@buggtb help

@dmalaescu
Please run bin/dockler.sh script to get bash inside sparkler's docker image.
It uses root user inside docker which has all the permissions (note: root user is inside docker, it is not the same as root user in your host OS, so it can't do any harm to your host operating system)

thanks @thammegowda

@thammegowda
Thanks, it worked. I've managed to complete the steps with root

@dmalaescu
I am glad it worked.
FYI I fixed the issues and released a new docker image to dockerhub.
Next time you want to use sparkler, just do
docker run -it -p 8983:8983 -p 4040:4040 uscdatascience/sparkler

CC @chrismattmann

thanks great work @thammegowda


⚠ Bofrost.de has degraded performance

In f1b4cc2, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1268 ms

Resolved: Bofrost.de performance has improved in a32ad5b after 40 minutes.


Add ETH rate to wallet

Please add ETH rate to settings and wallet itself https://zpl.io/aXA0vLE

added beam-eth rate to rate notification script

checked


Pill component

@tomgenoni — is this ready for review? Feel free to make it and add me as a review. Looks good to me. I also made sure that this could work with Tooltip and it works fine if you have Tooltip > TextButton > Pill.


JSON::ParserError 784

This is probably easy, but I cannot get it to work...
I am fetching a JSON from an IOT. As a result, I get this:
  {     "power": 14.627669334411621,     "relay": true,     "temperature": 20.437118530273438   }
I am trying to pass only the "temperature" tag to a peak detector agent, by using a Json Parse Agent.
The Json Parse Agent has "{{temperature}}" under Data field and "data" under Data key field, but I am always getting this message in the logs:
{   "data": "{{temperature}}",   "data_key": "data" }
Could not parse JSON: JSON::ParserError '784: unexpected token at '20.37446403503418''
Any help would be appreciated.

That JSON looks valid to me, can you post the full Event payload that is send to the JsonParseAgent from either the Events or the source Agents detail page?

Thanks for your kind help. Here is some additional info:

Ah! You don't need the JsonParseAgent. The JSON has already been parsed by the WebsiteAgent, you can access the temperature in the "Hystou hot" Agent via {{temperature}}.

Excellent ! Many thanks !!!


Update steampipe plugin SDK version to 2.1.1

Updated in https://github.com/turbot/steampipe-plugin-gcp/pull/347


Charger disabled: vehicle not capable of start/stop when nothing connected to chager

Describe the bug
Since the update to v0.119.4 I keep getting an notification saying the charger disabled: vehicle not capable of start/stop. At this moment there is nothing connected to my tesla wall charger.

Steps to reproduce

update to v0.119.4
open webpage
see notifications coming in.
...

Configuration details
# open evcc at http://evcc.local:7070
network:
  schema: http
  host: evcc.local # .local suffix announces the hostname on MDNS
  port: 7070

log: debug
levels:
  cache: error

# unique installation id
plant: xxxxx

interval: 10s # control cycle interval

meters:
- name: pv2
  type: custom
  power:
      source: http
      uri: http://192.168.1.xxx/e
      method: GET # default HTTP method
      headers:
        - content-type: application/json
      insecure: true # set to true to trust self-signed certificates
      jq: .[0].ps0 # parse response json
      scale: 1.0 # floating point factor applied to result, e.g. for kW to W conversion
      timeout: 10s # timeout in golang duration format, see https://golang.org/pkg/time/#ParseDuration
- name: dsmr
  type: custom
  power:
    source: calc
    add:
    - source: http
      uri: http://192.168.1.xxx/api/v2/consumption/electricity-live
      method: GET # default HTTP method
      headers:
      - content-type: application/json
      - X-AUTHKEY: xxx
      insecure: true # set to true to trust self-signed certificates
      jq: .|{currently_delivered}| join(" ")
      scale: 1
      timeout: 10s
    - source: http
      uri: http://192.168.1.xxx/api/v2/consumption/electricity-live
      method: GET # default HTTP method
      headers:
      - content-type: application/json
      - X-AUTHKEY: xxx
      insecure: true # set to true to trust self-signed certificates
      jq: .|{currently_returned}| join(" ")
      scale: -1
      timeout: 10s
  energy:
    source: calc
    add:
    - source: http
      uri: http://192.168.1.xxx/api/v2/consumption/today
      method: GET # default HTTP method
      headers:
      - content-type: application/json
      - X-AUTHKEY: xxx
      insecure: true # set to true to trust self-signed certificates
      jq: .|{electricity_merged}| join(" ")
      scale: 1
      timeout: 10s
    - source: http
      uri: http://192.168.1.xxx/api/v2/consumption/today
      method: GET # default HTTP method
      headers:
      - content-type: application/json
      - X-AUTHKEY: xxx
      insecure: true # set to true to trust self-signed certificates
      jq: .|{electricity_returned_merged}| join(" ")
      scale: -1
      timeout: 10s
  currents:
  - source: http
    uri: http://192.168.1.xxx/api/v2/datalogger/dsmrreading?limit=1&ordering=-timestamp
    method: GET # default HTTP method
    headers:
    - content-type: application/json
    - X-AUTHKEY: xxx
    insecure: true # set to true to trust self-signed certificates
    jq: .results[]|{phase_power_current_l1}| join(" ")
    scale: 1
    timeout: 10s
  - source: http
    uri: http://192.168.1.xxx/api/v2/datalogger/dsmrreading?limit=1&ordering=-timestamp
    method: GET # default HTTP method
    headers:
    - content-type: application/json
    - X-AUTHKEY: xxx
    insecure: true # set to true to trust self-signed certificates
    jq: .results[]|{phase_power_current_l2}| join(" ")
    scale: 1
    timeout: 10s
  - source: http
    uri: http://192.168.1.xxx/api/v2/datalogger/dsmrreading?limit=1&ordering=-timestamp
    method: GET # default HTTP method
    headers:
    - content-type: application/json
    - X-AUTHKEY: xxx
    insecure: true # set to true to trust self-signed certificates
    jq: .results[]|{phase_power_current_l3}| join(" ")
    scale: 1
    timeout: 10s

chargers:
- type: template
  template: twc3
  host: <IP_ADDRESS>
  name: wallbox3

loadpoints:
- title: Oprit
  charger: wallbox3
  mode: off
  phases: 3
  mincurrent: 2
  maxcurrent: 16
  resetOnDisconnect: false

site:
  - title: Huis
    meters:
      grid: dsmr # grid meter reference
      pv: # (pvs = veraltet)
        - pv2  # first pv meter reference
vehicles:
  - name: my_car
    type: template
    template: tesla
    title: Executor # Wird in der BenutzeroberflÃ¤che angezeigt (Optional)
    accessToken: xxxx
    refreshToken: xxx
    vin: xxx
    #onIdentify: # set defaults when vehicle is identified
    mode: pv # enable PV-charging when vehicle is identified
    minSoc: 30 # immediately charge to 0% regardless of mode unless "off" (disabled)
    targetSoc: 90 # limit charge to 90%

tariffs:
  currency: EUR # three letter ISO-4217 currency code (default EUR)
  grid:
    type: tibber
    token: "xx"
  # grid:
    #either static grid price (or price zones)
    # type: fixed
    # price: 0.58 # EUR/kWh
    # zones:
      # - days: Mo-Fr
        # hours: 07-21
        # price: 0.73 # EUR/kWh
      # - days: Sa,So
        # price: 0.58 # EUR/kWh
mqtt:
  broker: <IP_ADDRESS>:1883
  topic: evcc # root topic for publishing, set empty to disable publishing
  # clientid: foo
  user: evcc
  password: xxxx

Log details
root@evcc:~# evcc --log debug
[main  ] INFO 2023/09/04 11:32:36 evcc 0.119.4
[main  ] INFO 2023/09/04 11:32:36 using config file: /etc/evcc.yaml
[main  ] INFO 2023/09/04 11:32:36 starting ui and api at :7070
[db    ] INFO 2023/09/04 11:32:36 using sqlite database: /root/.evcc/evcc.db
[mqtt  ] INFO 2023/09/04 11:32:36 connecting evcc-836473421 at tcp://<IP_ADDRESS>                                                                                :1883
[mqtt  ] DEBUG 2023/09/04 11:32:36 tcp://<IP_ADDRESS>:1883 connected
[site  ] INFO 2023/09/04 11:32:37 site config:
[site  ] INFO 2023/09/04 11:32:37   meters:      grid ✓ pv ✓ battery ✗
[site  ] INFO 2023/09/04 11:32:37     grid:      power ✓ energy ✓ currents ✓
[site  ] INFO 2023/09/04 11:32:37     pv 1:      power ✓ energy ✗ currents ✗
[site  ] INFO 2023/09/04 11:32:37   vehicles:
[site  ] INFO 2023/09/04 11:32:37     vehicle 1: range ✓ finish ✓ status ✓ clima                                                                                te ✗ wakeup ✓
[lp-1  ] INFO 2023/09/04 11:32:37 loadpoint 1:
[lp-1  ] INFO 2023/09/04 11:32:37   mode:        off
[lp-1  ] INFO 2023/09/04 11:32:37   charger:     power ✓ energy ✗ currents ✓ pha                                                                                ses ✗ wakeup ✗
[lp-1  ] INFO 2023/09/04 11:32:37   meters:      charge ✓
[lp-1  ] INFO 2023/09/04 11:32:37     charge:    power ✓ energy ✗ currents ✓
[lp-1  ] DEBUG 2023/09/04 11:32:37 phase timer inactive
[lp-1  ] DEBUG 2023/09/04 11:32:37 pv timer inactive
[lp-1  ] DEBUG 2023/09/04 11:32:37 guard timer inactive
[site  ] WARN 2023/09/04 11:32:37 interval <30s can lead to unexpected behavior,                                                                                see https://docs.evcc.io/docs/reference/configuration/interval
[site  ] DEBUG 2023/09/04 11:32:37 ----
[lp-1  ] DEBUG 2023/09/04 11:32:37 charge power: 0W
[site  ] DEBUG 2023/09/04 11:32:37 pv power: 5741W
[site  ] DEBUG 2023/09/04 11:32:37 grid power: -5201W
[site  ] DEBUG 2023/09/04 11:32:37 grid currents: [6 7 7]A
[site  ] DEBUG 2023/09/04 11:32:37 site power: -5201W
[lp-1  ] DEBUG 2023/09/04 11:32:37 charge voltages: [0 0 0]V
[lp-1  ] DEBUG 2023/09/04 11:32:37 charge currents: [0.3 0 0.4]A
[lp-1  ] DEBUG 2023/09/04 11:32:37 charger status: A
[lp-1  ] INFO 2023/09/04 11:32:37 car disconnected

What type of operating system are you running?
Linux
Version
evcc version 0.119.

Could you please add a log of the actual problem with
twc: trace
tesla: trace

will reopen when log available.

Sorry got same problem and even has never worked at all. Do I have to get token each day, each start. Neverless, I got connection with Y and Tesla wall connector. But there was no interaction.
[site  ] DEBUG 2024/02/06 20:29:40 ----
[lp-1  ] DEBUG 2024/02/06 20:29:40 charge power: 1824W
[site  ] DEBUG 2024/02/06 20:29:40 pv power: 0W
[site  ] DEBUG 2024/02/06 20:29:40 site power: 2224W
[lp-1  ] DEBUG 2024/02/06 20:29:40 charge voltages: [225 2.5 0]V
[lp-1  ] DEBUG 2024/02/06 20:29:40 detected connected phases: 1p
[lp-1  ] DEBUG 2024/02/06 20:29:40 charge currents: [8.1 0 0.5]A
[lp-1  ] DEBUG 2024/02/06 20:29:40 detected active phases: 1p
[lp-1  ] DEBUG 2024/02/06 20:29:40 charger status: C
[lp-1  ] DEBUG 2024/02/06 20:29:40 vehicle soc: 68%
[lp-1  ] DEBUG 2024/02/06 20:29:40 vehicle soc limit: 80%
[lp-1  ] DEBUG 2024/02/06 20:29:40 vehicle range: 284km
[lp-1  ] DEBUG 2024/02/06 20:29:40 pv charge current: 0A = 0A + -9.67A (2224W @ 1p)
[lp-1  ] DEBUG 2024/02/06 20:29:40 site power 2224W >= 0W disable threshold
[lp-1  ] DEBUG 2024/02/06 20:29:40 pv disable in 0s
[lp-1  ] DEBUG 2024/02/06 20:29:40 pv disable timer elapsed
[lp-1  ] ERROR 2024/02/06 20:29:40 charger disable: vehicle not capable of start/stop
[site  ] DEBUG 2024/02/06 20:29:50 ----
[lp-1  ] DEBUG 2024/02/06 20:29:51 charge power: 1826W
[site  ] DEBUG 2024/02/06 20:29:51 pv power: 0W
[site  ] DEBUG 2024/02/06 20:29:51 site power: 2226W
[lp-1  ] DEBUG 2024/02/06 20:29:51 charge voltages: [225 2.5 0]V
[lp-1  ] DEBUG 2024/02/06 20:29:51 detected connected phases: 1p
[lp-1  ] DEBUG 2024/02/06 20:29:51 charge currents: [8.1 0 0.5]A
[lp-1  ] DEBUG 2024/02/06 20:29:51 detected active phases: 1p
[lp-1  ] DEBUG 2024/02/06 20:29:51 charger status: C
[lp-1  ] DEBUG 2024/02/06 20:29:51 vehicle soc: 68%
[lp-1  ] DEBUG 2024/02/06 20:29:51 vehicle soc limit: 80%
[lp-1  ] DEBUG 2024/02/06 20:29:51 vehicle range: 284km
[lp-1  ] DEBUG 2024/02/06 20:29:51 pv charge current: 0A = 0A + -9.68A (2226W @ 1p)
[lp-1  ] DEBUG 2024/02/06 20:29:51 site power 2226W >= 0W disable threshold
[lp-1  ] DEBUG 2024/02/06 20:29:51 pv disable in 0s
[lp-1  ] DEBUG 2024/02/06 20:29:51 pv disable timer elapsed
[lp-1  ] ERROR 2024/02/06 20:29:51 charger disable: vehicle not capable of start/stop


Send message to some groups

Hi, i am new to this topic and this language, i'm trying to send the exctly same message at the exctly same time to 3 different groups in skype, im out of ideas how to do it, even chatgpt coldn't  help me out. By now, i have this code below.
`from skpy import Skype,
import requests, time, schedule
Substitua 'seu_email' e 'sua_senha' pelas suas credenciais do Skype
sk = Skype("...", "m...")
Lista de IDs de contatos para os quais você deseja enviar a mensagem
contacts_ids = ['live:.cid.6000b2330b1a104c', 'live:.cid.a84ba69c28c5bfbd']
Loop para enviar a mensagem para cada contato
for contact_id in contatos_ids:
# Encontre o contato pelo ID
contact = sk.contacts[contacts_ids]
def send_message():
# Coloque aqui o código para enviar a mensagem
message = "Sua mensagem aqui"  # Substitua pela mensagem que você deseja enviar
contact.chat.sendMsg(message)
Agende o envio da mensagem às 15 horas todos os dias
schedule.every().day.at("15:00").do(send_message)
O loop a seguir verifica regularmente se há tarefas agendadas para serem executadas
while True:
schedule.run_pending()
time.sleep(1)  # Você pode ajustar o intervalo de verificação, se desejar`

You haven't said what your problem is, and the one bit of SkPy code (sk.contacts[contact_id].chat.sendMsg(message)) should work, so without any more context, either there's a problem elsewhere in your code or you're getting rate-limited from spamming your contacts every second if run_pending() actually triggers the sending every second.

Im getting de error 404

This is acturaly a group, i'm trying to send a message to three different groups

The code you posted originally does not create or interact with any groups, it's sending messages to individual users.  Please post the code you're actually running code in full, and show the corresponding output and errors.
Peeking at that traceback, that group ID doesn't look long enough.  Are you passing a user ID instead of a group chat ID?  You should also be using key lookups on sk.chats, no need to call chat().

I don't even know if this makes sense at all, i'm new at dev and just trying this

No problems here reading a group like that:
>>> group_id =<EMAIL_ADDRESS>>>> group = sk.chats.chat(group_id)
>>> group
<EMAIL_ADDRESS>alerts=True, topic='Test group', creatorId='8:fred.2', userIds=['8:fred.2'], adminIds=['8:fred.2'], open=False, history=False, active=False, moderated=False)

Or using key access as recommended earlier:
>>> sk.chats[group_id]
<EMAIL_ADDRESS>alerts=True, topic='Test group', creatorId='8:fred.2', userIds=['8:fred.2'], adminIds=['8:fred.2'], open=False, history=False, active=False, moderated=False)

Reiterating from above:

Peeking at that traceback, that group ID doesn't look long enough. Are you passing a user ID instead of a group chat ID?

The prefix (19:) is correct so I'm unsure if you've actually obtained it from somewhere, or just manually typed an invalid ID.
sk.contacts[group_id] (line 21) also won't work, presumably you mean to use group there.
As far as I can tell, there's no issue with SkPy here, so please take some time to read the Guides section of the docs that explains how to do the basic tasks like retrieving users and chats.

Bro, your tip about de

The code you posted originally does not create or interact with any groups, it's sending messages to individual users. Please post the code you're actually running code in full, and show the corresponding output and errors.
Peeking at that traceback, that group ID doesn't look long enough. Are you passing a user ID instead of a group chat ID? You should also be using key lookups on sk.chats, no need to call chat().

This tip solved my problems, thank you so much


chore: run poetry lock & add poetry-check to CI

Summary

Minor change to fix the incorrect poetry.lock file and add a poetry-check to the pre-commit action.
Motivation

Fix previous mistake and prevent if from happening again.
Author Checklist

[x] Changes are motivated and described in the PR description.
[x] Testing strategy is described if automated tests are not included in the PR.
[x] Documentation is updated if necessary.
[x] Status checks pass.
[x] If the version number is changed, it is updated in pyproject.toml , aineko/__init__.py, aineko/templates/first_aineko_pipeline/{{cookiecutter.project_slug}}/pyproject.toml.

Reviewer Checklist

[ ] Title is accurate and formatted appropriately.
[ ] No unnecessary changes are introduced.
[ ] Description motivates each change.
[ ] Docs are updated if necessary.
[ ] Code is pythonic and consistent with the rest of the codebase.
[ ] The code is consistent with the style guide.

LGTM, I'm rolling this with the version bump PR so closing this


How to specify GPU

When I have two GPUs, how can I specify which GPU to choose when calling methods?
e.g. Deepface.find()
I checked the parameters of this method, and it seems that there's no parameter specified for GPU. I'm not sure if deepface has a configuration only for GPU.
thanks.

Set cuda visible devices argument with gpu id

can you give full command example?
etc modify this :   result = DeepFace.verify(pair[0], pair[1], model_name=model_name)


append_pdfs error

Hello, I continue to get the following error any time I try to use append_pdfs:
Error using append_pdfs (line 131)
Unterminated quote in @-file: temp.pdf"
where temp.pdf is whatever the last file was in the list of files that I was trying to append. This has been happening on and off for months now but usually resolves when I unlink and then relink Ghostscript with Homebrew. However, this did not work this time. I have the most updated Ghostscript and the most updated export_fig. Please advise.
Here are 2 sample inputs and the error they throw:

append_pdfs('final.pdf', 'temp1.pdf', 'temp2.pdf', 'temp3.pdf')
Error using append_pdfs (line 131)
Unterminated quote in @-file: temp3.pdf"
saveas(gcf, append('temp.pdf'))
append_pdfs(pdf_name, append('temp.pdf'))
Error using append_pdfs (line 131)
Unterminated quote in @-file: temp.pdf"

Please try it with the latest commit (694e85b) and let me know if it solved the problem or not.
If the problem persists, please send me the extra text output that should not be displayed in case of a ghostscript error, that will help debug the problem.

Unfortunately, that did not fix the issue. The extra text + error now looks like this:
-q -dNOPAUSE -dBATCH -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress -sOutputFile="/private/var/folders/03/7r1dhlhd0ds9y8t_9cpzgqrr0000gn/T/tp1b76df99_d9e7_4796_9244_9c6417bffd76.pdf" -f "RESULTS_2023-06-11 13-38_PPlnth-4_with_int.pdf" "temp.pdf"
Error using append_pdfs (line 134)
Unterminated quote in @-file: temp.pdf"

Please check if the latest commit (47ab1f9) fixes this issue.
In this latest version, there should only be quotation marks now around filenames that contain a space, and hopefully this resolves the problem.


CLN: Split base API

Coverage increased (+0.8%) to 97.819% when pulling acd4034d7c305e595275fc00c9637a58159aeb76 on api_cln into 14ad94cc2fcc773f9002196b7d72bbeb47fa3ce7 on master.

Coverage increased (+0.8%) to 97.819% when pulling acd4034d7c305e595275fc00c9637a58159aeb76 on api_cln into 14ad94cc2fcc773f9002196b7d72bbeb47fa3ce7 on master.


Github badges are all inaccessible

I'm not sure whether this is due to one of the recent changes…
#1142 #1117 #1195 #1186 #1118
…or simply #1119, which is a bug that causes a token to erroneously be considered exhausted once it's used for a search request.
People can't add new tokens either (#1243), exacerbating this slightly, but that will be fixed in #1038.
The first report was roughly 16 hours after deploy.

cc @espadrine

The badges are working again, and I think our "main" rate limit just reset:
core:
  remaining: 12489 of 12500
  reset: in an hour
search:
  remaining: 30 of 30
  reset: in a minute
graphql:
  remaining: 5000 of 5000
  reset: in an hour

Now intermittently broken, though plenty of rate limit left.
core:
  remaining: 12479 of 12500
  reset: in 18 minutes
search:
  remaining: 30 of 30
  reset: in a minute
graphql:
  remaining: 5000 of 5000
  reset: in an hour

Here's an example: https://img.shields.io/github/tag/expressjs/express.svg

I sent this to @espadrine about an hour ago:

The Github badges are failing intermittently. Are you seeing crashes on the server?
It might be an old bug, related to our handling of quotas for the Github search API. Though the timing of the incident makes me suspect recent changes. I've made some recent changes to the github auth but it's not jumping out at me from reading them.
It's difficult to debug without server access. I'm thinking I should add an endpoint to get all the user tokens, or else, hashed user tokens with stats. That way I could troubleshoot a bit better, locally.
Is there currently any backup of the user tokens, apart from the other servers?
I do have some new github token code to fix the search API quota issue, though it's a rewrite and I'd like to test it more first. Before merging I also want to add some optional trace logging we can turn on in cases like this.
Feel like I need deploy access + logs + a way to restore the token file, to deploy that with confidence I can find and fix whatever might be wrong with it.
Any thoughts on what could be causing the ssh issue?

I like getting to the bottom of things, and want to fix this, however my options are limited.

I set up a status page:
https://status.shields-server.com/
It runs a static badge, the Github license badge, and the npm license badge, and for each one, looks for some of the expected right text.

Nice page, Should help get some insight on what's going wrong,
The GitHub licence badge seems to be failing a fair amount (~20% currently)
Is s1, s2, s3 running different code or are they all the same?

Yea, it should help. The code on the three servers should be the same.
There are interesting patterns in the downtime:
https://status.shields-server.com/779605524
https://status.shields-server.com/779605526
https://status.shields-server.com/779605529
The three servers had correlated downtime around 15:30 (that’s NY time). One of them also had downtime an hour earlier, around 14:30. Two had downtime around 13:33 / 13:43.
The duration of the downtime varies from server to server. For example, s0 was down from 15:28 to 15:49, s1 from 15:30 to 15:36, and s2 from 15:32 to 15:48.
Correlated downtime suggests there is some shared state, pointing to rate limit exhaustion as a factor. Downtime about an hour apart might correlate with rate limit resets.
The skew in recovery time might be explained by caching, though there might be other explanations too.

Yeah it's quite strange that the down times are very similar, would setting a very low max-age help with possible caching issues?

As far as I can tell, maxAge only affects cache headers – and potentially the behavior of the client – though not the behavior of the Shields server. I wouldn't think UptimeRobot did any caching. It wouldn't really make sense for a monitoring service. So I don't think setting maxAge would have any effect.

I just wanted to clarify that the caching I think might be involved is the Shields internal vendor cache in lib/request-handler.js.
Interesting that we're still seeing hourly downtime, though less correlated between servers. I wonder if it is related to hours since uptime.

Still seems to be failing ~20% of the time,
Any clues yet as to what the problem could be?
It still seems they generally go down/up within 5-20 minutes of each other.

Three things.
s1 is European (located at Gravelines, IIRC), while s0 and s2 are in Canada (Montreal?).
Most of our vendors (typically, GitHub) have US servers.
It is inevitable that crossing the Atlantic yields a poorer SLA.
On the plus side, it is the least infuriating SSH session for me, and Europeans enjoy a faster static badge thanks to it.
Second, the worldwide load looks like this.

I can't recall what the third thing was, but maybe it was related to describing exactly the shape of the failures? Like, is it failing once every ten during the high-load hour?

Just bumping to say I experience non-loading badges for several days now. Every other refresh I get Invalid upstream response (521) from githubusercontent.com

I've been seeing a lot of this the last few days:

Indeed, this has happened with a good chunk of requests over the last few days.
https://status.shields-server.com/
Things have been much worse the last 22 hours because of #1263, which is unrelated service-provider downtime that took out one of our servers.

s1 is European (located at Gravelines, IIRC), while s0 and s2 are in Canada (Montreal?).
Most of our vendors (typically, GitHub) have US servers.

Good to know. That explains why the stats for s1 are sometimes slightly worse.

To re-summarize:

@espadrine, who has limited time on this project, is the only sysadmin.
He's working on giving me access.
Doing so is complicated because the hosting account (and maybe the servers too) are shared with other services he runs.
I like getting to the bottom of things, and want to fix this, however my options are limited.

I just emailed this plan:

To solve #1119, I rewrote the GitHub auth logic which is in an unmerged PR. There were other minor bugs I found along the way. A logic error in the token sorting, a missing callback.
I’d like to deploy that new code, but it’s a big change and I don’t feel comfortable doing it without some way to back up and restore the tokens, and deploy and logs access or else a deploy window when you’re around.
Here’s what I’ll do:

Add some debug output and/or debug API to the current github-auth code
Self-review, again, the new github-auth PR
Add debug output and/or debug API to the new github-auth PR

Could I ask you to:

Check how many tokens we have in production
Deploy latest so we can start collecting additional tokens (it’ll help a little, I think)
Sort out the logging
Debug the ssh issue

@paulmelnikow I was checking the links:
https://img.shields.io/codecov/c/github/bragful/ephp.svg
https://img.shields.io/travis/bragful/ephp/master.svg
They are working but they take too much time to load (around 15 seconds). Github works using a proxy to retrieve this kind of images so, the error retrieved by the browser is a 504 (Gateway timeout).
Did you check the amount of requests your system is receiving to generate the badges? If I can help you with something just let me know.

@manuel-rubio Yea, that's unfortunate. See #1263.

While working on

Add some debug output and/or debug API to the current github-auth code

I found the issue.  Fixed in #1266.
AFAICT production has been running with anonymous quota. I'm shocked this has been working as well as it has. Admittedly, not that well, though I'd have expected what we have to work for the first few seconds of every hour.
Either the server is using a different github secret from the one I expect, or as likely, the Shields IPs do indeed have special treatment from GitHub.
I'm still eager to get the new code shipped, as it has a lot more tests. And of course #1263 remains an issue.

Opened #1267 with an auth debug endpoint + logging.

If I can help you with something just let me know.

I didn't really answer this question @manuel-rubio!
There are four ways you can help:

Review my changes. I have to self-review my code, which is hardly ideal. For what it's worth, the PR that caused this regression was open for four weeks, which is plenty of opportunity. If a team of five people could review a couple PRs per week, my changes could easily have 2–3 reviews apiece. Not only does would this reduce bugs, over time it has a wonderful side effect of making the code more readable and therefore more approachable.
Perform first reviews of simple changes, like badge additions.
Monitor issues and the chat room, and help other people who have questions about contributing to Shields, or using it for their projects. Dig into the code as needed. This is the easiest way to create time among the people who have the most context on this project.
Contribute GitHub tokens and $. I honestly don't know much about our current financial state, though I would love to have the flexibility to use third-party monitoring and logging services, and not to mention choose hosting that makes scaling and shared administration easy. We're setting up an OpenCollective since Gratipay is shutting down.

The fix is deployed. Status looks good:


Add compound colors dict, minor docstring rewrite

I got an email that you assigned this issue to a milestone so I assumed you want it to be worked on.
The process is not very rigorous. I found the tyre icons' image files used on the F1 live timing page and got the color codes from there.

Soft: da291c
Medium: ffd12e
Hard: f0f03c
Intermediate: 43b02a
Wet: 0067ad

Sometimes the data will have the color compound as UNKNOWN (this column is never null). I am representing this as cyan which should be clearly distinguishable from the rest.

Rewrote the few similar docstrings since the old wording sounds a bit backward

Yes, I wanted this to be in the next release, but my intention was not to make you work on it 😂 Anyway, this looks good, so thank you!

Well I guess someone has to do it some time and I happened to have some time on my hands haha


Added authenticatorDisplayName to ClientPropertiesOutput

Based on the current editors draft on https://w3c.github.io/webauthn/ a new field authenticatorDisplayName has been added: https://w3c.github.io/webauthn/#dom-credentialpropertiesoutput-authenticatordisplayname
Thought it would be feasible to add this field, since it provides valuable information :)

:warning: Please install the  to ensure uploads and comments are reliably processed by Codecov.
Codecov Report
Attention: Patch coverage is 0% with 1 line in your changes missing coverage. Please review.

Project coverage is 74.46%. Comparing base (9ad038b) to head (608afd2).

Files
Patch %
Lines

...Fido2.Models/Objects/CredentialPropertiesOutput.cs
0.00%
1 Missing :warning:

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@            Coverage Diff             @@
##           master     #526      +/-   ##
==========================================
- Coverage   74.49%   74.46%   -0.03%
==========================================
  Files          98       98
  Lines        2639     2640       +1
  Branches      445      446       +1
==========================================
  Hits         1966     1966
- Misses        570      571       +1
  Partials      103      103

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

@dotnet-policy-service agree company="Aritma"


Can't find any info on TSCLIENT connections

The RDP app seems useless.  Menus are weird.  Can barely know if I've even 'saved' something.
There is no area about TSCLIENT connections.  Especially for workspaces.

Hey @ComputerHabit2,
Thanks again for contacting Microsoft, we value your feedback.
You have described an issue with a product, service, or feature request rather than our documentation. Please see our GitHub content feedback guidelines. As this is a non-documentation issue, we’ll now be closing this thread.
Thank you and have a most pleasant day.
#please-close


httproute add ipmatch and cookiematch

What would you like to be added:
httproute add ipmatch and cookiematch
Why this is needed:

Use cases for IP-based matching:

Geolocation routing: Route requests to specific geographic locations based on the user's IP address. For example, distribute requests to server clusters located in the user's country or region to provide a better localized experience.
Data center routing: Route requests to specific data centers for distributed deployment and load balancing. Distribute traffic to the nearest or most suitable data center based on the source or destination IP of the request.
Blacklisting/whitelisting: Intercept or allow access to requests from specific IP ranges or individual IP addresses based on their IP addresses. This can be used for network security policies or access control purposes.

Use cases for cookie-based matching:

Session affinity: Route requests to backend servers associated with a specific cookie from previous interactions to maintain consistent user sessions. This is important for applications that require session state to be preserved throughout a user's session.
A/B testing: Split users into different backend server groups or versions based on the cookies in the request to test different features or user experiences and collect comparative data for decision-making.
User authentication and authorization: Route users to the appropriate backend servers for authentication and authorization based on the cookies in the request. This allows separation of the authentication process from the handling of subsequent requests, enhancing security and performance.

### Tasks

So you mean adding new Matchers to HTTPRouteMatch?
Some implementations like Apache APISIX support these extra matches, but I am not sure it can go into the standard or allow implementation-specific extensions.

@sdjksdajshd since we haven't heard back from you on this one in some time, we're going to close this one for now.
/close
If you would still like to pursue these features, please create separate issues for each feature as Rob suggested in July.


Delete and Read Async methods

This will resolve #34 and #17

There are tests that I forgot to write. it got lost in the tediousness. so review this but know another commit will be coming


Better support for choice of endpoint for aioredis

Hello,
sorry for the delay.
As discussed in https://github.com/argaen/aiocache/issues/426#issuecomment-443124618
aiocache only seemed to allow for setting up the address and port and not the unix socket. I believe the issue is simple:
aioredis offers 3 choices from documentation:

a Redis URI — "redis://host:6379/0?encoding=utf-8"; "redis://:password@host:6379/0?encoding=utf-8";
a (host, port) tuple — ('localhost', 6379);
a unix domain socket path string — "/path/to/redis.sock".

but aiocache only supports one option:
self._pool = await aioredis.create_pool((self.endpoint, self.port), **kwargs)
I have tried removing the tuple with the port, and the unix connection worked so it should not be very difficult to fix this.
I am happy to submit a PR with the change needed to support all 3 options. The easiest way would probably be to offer exactly the same inputs as aioredis, but that would break the scheme with the other backends. Maybe we can assume that if the port is not specified or if the endpoint contains a slash, it is not the tuple option and pass the string only. Do you have a safer idea?
Thank you!

Hey @John-Gee thanks for opening this. Yes I've been thinking about this for a while, in the latest version of master there is this Cache.from_url available: https://github.com/argaen/aiocache/blob/master/aiocache/factory.py#L100
This relies on a method from each backend that can parse custom URIs for each backend implementation. For example the Redis one: https://github.com/argaen/aiocache/blob/master/aiocache/backends/redis.py#L255. I'm starting to think maybe it would be better if instead of just parsing the path, each cache could be responsible of the implementation of parsing the full url, then we could put this behavior you mention in that implementation...
I like this more because its more explicit than checking if there is a slash in the endpoint, what do you think? looking for a second opinion here :)

Probably because I don't have a good understanding of aiocache or aioredis, I am a bit lost here.
If you have a function that works on a URI you already know not to make a tuple out of it, so why do you need to parse the URI when, at least in redis' case, the db will accept the full string?
It should be fairly easy to see if what's passed is actually a redis scheme or simple socket path, and return the correct options object. Which does seem clean indeed.

If you have a function that works on a URI you already know not to make a tuple out of it, so why do you need to parse the URI when, at least in redis' case, the db will accept the full string?

Design decisions I did in the beginning, trying to fix it now somehow :). Backends have public endpoint and port attributes so will have to parse for this in case. But yes the underlying implementation can be changed so in case of Redis we forward the URL directly instead of using a tuple.
My point being: I don't want to do breaking changes in order to fix this, but yes it should be possible to fix this in a nice way :)

Fair enough.
I've just tested and simply not passing the port is not breaking, you just default to the standard one; so checking for that parameter would not work anyway.
Where would you like to parse the endpoint and make the decision? It would work in _get_pool I believe but I'm guessing it's not the optimal place. Although it's the function where we would need to know which call to make.

I would support this only in the Cache.from_url method and leave the constructors like RedisCache out of the equation. Then I would implement the functionality in the method parse_uri_path (renaming it of course) in order to decide if what we receive is a url or a unix socket.
I'm talking about current master version.
Does that make sense to you?

Are you saying that you want the constructors to not gain the new behavior but instead have it only through Cache.from_url?
If so, that'd be very fine from my little user's point of view, but it makes your API a little weird I think.
Now at the end of Cache.from_url you currently call the constructor, that eventually will call RedisBackend._get_pool. Does your plan involve changing that function or not calling it?

Are you saying that you want the constructors to not gain the new behavior but instead have it only through Cache.from_url?

yup

but it makes your API a little weird I think.

Well.. depends on how you look at it, you have the constructors that have the same behavior as before, where you receive endpoint and port. Then you have a from_url method in the Cache factory in case you want this behavior.
I want to move usage to always use Cache instead of the constructors directly, it gives a single point of entry

Now at the end of Cache.from_url you currently call the constructor, that eventually will call RedisBackend._get_pool. Does your plan involve changing that function or not calling it?

Yes I'm not 100% sure how to link that, would have to play a bit with it to get an idea. Feel free to make a proposal and we can review if you want :)

I want to move usage to always use Cache instead of the constructors directly, it gives a single point of entry

Oh if you're planning on getting there that's fine, it'll just be a bit weird during depreciation time.

Yes I'm not 100% sure how to link that, would have to play a bit with it to get an idea. Feel free to make a proposal and we can review if you want :)

Well, maybe the address object needed by aioredis could be created outside of that function, and passed to it. That way that function merely passes it without getting more complicated.
Instead of

self._pool = await aioredis.create_pool((self.endpoint, self.port), **kwargs)

You'd simply have

self._pool = await aioredis.create_pool(self.address, **kwargs)

with address either being (self.endpoint, self.port) or the url.
I'm not sure about having a member variable with a dynamic type though, it's a bit strange. Maybe having 2 variables, and simply using the non none one makes it clearer.
If I have time tonight, I'll try to write a prototype of this. If not I'll be going on vacation next week and wouldn't be able to get to it for about the next 2 weeks. Well unless I end up bringing a laptop, which I might.
Once we get something acceptable for Redis, we could move onto the other backends. For the simple one I guess throwing an unsupported error would be enough, a dict does not need a complex URL... For memcached, the documentation of aiomcache is pretty sparse, but the code makes me think that there is no other option but the current one, so again simply not supporting this may be enough. This means that Cache.from_url would only be used for Redis, but since you made it generic, I'm probably missing something. What is it? :)
Thank you!

with address either being (self.endpoint, self.port) or the url.

if we get endpoint and port, we could still create the address manually in the constructor right? This way address always has the same type

This means that Cache.from_url would only be used for Redis, but since you made it generic, I'm probably missing something. What is it? :)

the from_url lets you specific any parameters, so for example if for the memory one you want to specify a namespace, you can do that providing memory://?namespace=hello. For memcached, you can do memcached://endpoint:port?timeout=1. One thing is what the underlying client supports, and the other is how we parse the input URL. We are just providing a standardised way of specifying initialisation of new instances and it happens that aioredis supports this out of the box by just forwarding the address while with the others, we still have to go with providing endpoint/port or tthe constructor args :)


Investigate custom actions

Some of our actions/jobs are becoming quite large and unweildy.  Look to see which can be wrapped in custom actions and simplified.  Action creation can go in a future issue.
Benefit:
To allow for changes on the Greenfield template to be shared to other repos using the template, actions can be created and shared or published with other teams.
Version numbers will be needed and teams will be expected to pull versions action if they have a need for it.
Greenfield pipeline need the permission to publish actions, conversations with Nick Corcoran.

Custom actions aren't currently allowed by BCGov.  We've settled on workflow calls in the helpers repo, so maybe that isn't such a big deal?  Any blockers are still TBD.


Roll Flutter from 2ed7cdf35a0e to 9b818d2b707f (3 revisions)

https://github.com/flutter/flutter/compare/2ed7cdf35a0e...9b818d2b707f
2022-03-10<EMAIL_ADDRESS>Update bug number for android_semantics_integration_test (flutter/flutter#99842)
2022-03-10<EMAIL_ADDRESS>Update dependencies (flutter/flutter#99874)
2022-03-10<EMAIL_ADDRESS>Roll Engine from 482218b6770e to 6239bfb88423 (2 revisions) (flutter/flutter#99929)
If this roll has caused a breakage, revert this CL and stop the roller
using the controls here:
https://autoroll.skia.org/r/flutter-plugins
Please CC<EMAIL_ADDRESS>on the revert to ensure that a human
is aware of the problem.
To file a bug in Plugins: https://github.com/flutter/flutter/issues/new/choose
To report a problem with the AutoRoller itself, please file a bug:
https://bugs.chromium.org/p/skia/issues/entry?template=Autoroller+Bug
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

Commit queue failed; closing this roll.


VISUALS (SPECS): Map Statuses & Types to iconography + create iconography (Part 1 of 2)

Continuation of work started in ALM - VISUALS: Define Draft of Default Icons for Work Items & VISUALS: Vet & Finalize Default Icons

Gather list of all Work Item Type icons, from the Issue Tracking & Scrum process templates
Map any existing icons to the statuses (We already have Epic, Bug)
Create any new icons for the other statuses (Working on getting definitions, will add them here)

Icons needed (according to Todd, 2 templates still outstanding).
See "Type 1" column in tabs:

"Issue Tracking WILTs"
"SCRUM WILTs":
https://docs.google.com/a/redhat.com/spreadsheets/d/1iQ3MBitXy5zWAvxWzHQKtXuv5SWs_uUCQvrqngfm5K8/edit?usp=sharing

VC's

All Work Item Type icons have been accounted for
All icons have been approved by stakeholders
Icons are in a format for consumption by development

@jryhanyc I received a request from the Dev team this evening for a list of all of the icons, what they belong to and their CSS classes. Would you be able to fold that request into this work? The request issue is here: https://github.com/fabric8io/fabric8-ux/issues/294.
It was also pointed out that in the first pass of the Work Item Status icons, the checkmark was used for 'Done', but is also used as the icon for the Confirm Button. Can you verify that that is the correct usage? Thanks!

@jryhanyc I just added the current state of the world in the 294-Icon Documentation comment thread. This list was derived from e-mail threads and cross-referencing those with the spreadsheet. I took a first stab at some definitions for a few of the Work Item Types, but need feedback to be sure they're accurate. A little progress, but nothing that will let you start the Work Item Types work yet. I'll update that list as things get verified.

Rollup of current (Unconfirmed) types & states:
*Notes:

"/" means the same icon could be used for both
Text in "()" are definitions - some need verification

Work Item Types (design handled by @jryhanyc)
Confirmed icons that are currently missing:

Feature
Issue (Confirmed by Max: something that has been observed to either be a problem or missing thing in the already running system?)
Papercut (Pending definition confirmation from Todd: "trivial to fix, but annoying bugs")
Scenario (Confirmed: a specific defined interaction/sequence of interactions with a system to achieve a given goal)
Task (Confirmed by Max: something performed in order to complete a work item where a work item can be a story, feature, backlog item, etc.)

Currently have proposed Icons:

Bug
Epic
Experience
Fundamental
Product Backlog Item (Not sure if Issue can also use this)
Value Proposition

States (design handled by @bdellasc)
Icons in bold are currently missing:

New
Approved (Confirmed: approved by PM or Ready-Ready, and can planed or handed off the the team.)
Committed (Confirmed: team has planned to work on it)
Active/In Progress
Resolved/Done
Closed (Confirmed: verified and meets the Definition of Done)**
Removed (Leave name as is for now)**

@bdellasc can you link in what you demoed today?

Icons verified with Todd, and will be tested:
https://redhat.invisionapp.com/share/N8AZVB493


Support for Android artifacts in release pipelines

We need to add support for Android artifacts in the Java release pipelines. This will probably mean we go back to the old PowerShell script rather than the YAML script fragments we have today. The reason we need this is that we are going to start seeing Maven artifacts with an *.aar instead of a JAR file coming through the pipeline.

Closing this, will be handled as part of the work in the Android repo, doesn't need to be here.


Positional arguments

It would be nice to improve a bit the handling of positional arguments to be
more unix-y. A practical example of what I'd have in mind:
cxxopts::Options options(argv[0], " - example command line options");
options.add_options()
  ("f,flag", "desc")
  ("input", "", cxxopts::value<string>(input), "INPUT")
  ("output", "", cxxopts::value<string>(output), "OUTPUT");
opts.parse(argc, argv, {"input", "output"});

Where:

the empty description suppresses the help for that option
the third argument to Options::parse is a list of positional arguments.
Just a vector<string>, which is moved to m_positional (now a vector<string>
as well).
short usage also iterates though m_positional to find argument names for
consistent short help.
parse() finally consumes positional arguments in a simple loop.
If a vector<> is encountered, it just eats all the remaining arguments.
parse_positional() is now just a compatibility stub that adds an element
to m_positional, although I find it less clear than the above behavior.

This should result in the following help:
Usage:
  ./example [OPTION...] INPUT OUTPUT - example command line options

  -f, --flag desc

With no changes to the public interface (and minimal source changes as well).

This sounds pretty reasonable. Are you suggesting that you would be able to make the changes?
Just so I understand, you want to link positional arguments to different named arguments? At the moment I think it just puts everything into one positional argument. So in the above example, parsing the first positional argument lets me get it as options.get['input'] and the second as options.get['output'].
I'm not sure what you mean by eating the remaining arguments. Do you mean that if one of the positional arguments takes a vector then it just gets all the rest of the positional arguments?

On 10/09/15 06:38, jarro2783 wrote:

This sounds pretty reasonable. Are you suggesting that you would be able
to make the changes?

I would be, though not now. I was looking at the code too see if this
would be possible, and it looks like.

Just so I understand, you want to link positional arguments to different
named arguments? At the moment I think it just puts everything into one
positional argument. So in the above example, parsing the first
positional argument lets me get it as |options.get['input']| and the
second as |options.get['output']|.

Yes.

I'm not sure what you mean by eating the remaining arguments. Do you
mean that if one of the positional arguments takes a vector then it just
gets all the rest of the positional arguments?

Right. It's pretty common for me to have:
$ cmd [flags] cmd file [file ...]
With the above, if you link "file" to vector, you effectively
get all the final positional arguments.
Granted, you can do the same without, but I think it would be a nice touch.

I don't think it's hard. I'm not really in a position to do it right now either. We can leave this issue open and someone can do it eventually. Let us know if you feel like doing it before I do.

I'm working on this now.

This is done now.

I am looking for functionality described by OP and it doesn't seem to work. Did smth change since this was implemented? Is there any other way to achieve it?


feat: add  field to Google Autocomplete constructor option type

I forgot to paste the field name into the commit message, its 'origin'.. should I create a new PR?


Re-authenticate if current token is expired

It seems commands just return nothing if the current token is expired, which is a bad UX in my opinion.
I would expect a silent re-auth if possible (using refresh tokens) or a new interactive browser sign-in.

I wanted to add a bit more context.
I've just used the following command: spacectl stack list
The JWT sent to the /graphql endpoint looked like this:
{
  "cip" : "foo",
  "aud" : [
    "https:\/\/foo.app.spacelift.io"
  ],
  "eml"<EMAIL_ADDRESS>  "iat" :<PHONE_NUMBER>.4954071,
  "nbf" :<PHONE_NUMBER>.4954081,
  "sub"<EMAIL_ADDRESS>  "tms" : [
    "admin"
  ],
  "exp" :<PHONE_NUMBER>.4954071,
  "avt" : "foo",
  "adm" : true,
  "psa" : "01GBCHJGJB4Z7J0GXZ5PXTTKFB",
  "iss" : "saml",
  "jti" : "01GBCHJGJ78S91A9G38TTHG2XA"
}

As per the nbf and exp values the JWT was valid from Friday, 26. August 2022 07:41:27 till Friday, 26. August 2022 17:41:27.
The command was run today, 28. Sept.
This is the response from the API:
{
  "data": {
    "stacks": []
  }
}

I would expect a "token expired" error message (401?), a reauth with either a cached refresh token OR a re-auth request/popup.


Let source backfill finish when there's no data from upstream

Current status and the problem
Previous algorithm in https://github.com/risingwavelabs/rfcs/pull/72:

We only track backfill_offset, but not upstream_offset, and we finish backfill only when upstream_offset > backfill offset. So if there's no data from upstream, we cannot finish backfill.
This wasn't a problem until we want blocking DDL on source (https://github.com/risingwavelabs/risingwave/issues/15587). The DDL will be blocked forever until new messages come, which would confuse users.
Solution
Since the message stream is unbounded, we have to do such kind of "compare progress" between backfill and upstream stream. But we can improve that by checking in the backfill side instead of the upstream side. The former is more controllable in the SourceBackfillExecutor.

We add a new memory state target_offset, which is updated to upstream_offset when upstream messages come. Then we can compare progress and finish backfill in the backfill side. (To be more precise, we can only enter SourceCachingUp stage and wait for upstream_offset to be larger than it.) After this, we still cannot finish backfill when there's no upstream message, but it may end slightly faster.
Initialize target_offset to the latest available offset instead of None (in Kafka this is high_watermark-1). With this, we can finish backfill even there's no message from upstream. Note that this will relies on the connector to providing the ability to fetch latest_offset.
There's still a case not covered: there's no history data to backfill at all. We still cannot finish backfill in this case. To optimize, we need the ability of has_message_available provided by the connector. And check this before starting backfill.
Perhaps we can also use has_message_available as an exit condition of backfill, but that may require larger change in the source reader.

Diagram of new algorithm:
(Note: this applies to each split. After https://github.com/risingwavelabs/risingwave/issues/18300, we do not need special hacks (wait for all splits finished to finish backfill), and can treat all splits equally.)
stateDiagram-v2
    state Init <<choice>>
    [*] --> hasMessageAvailable
    hasMessageAvailable --> Init
    Init --> Backfilling: Yes (Or Unknown)
target_offset #colone; fetchLatestOffset() (Or None)
    Init --> Finished: No
    state Backfilling {
        [*] --> backfilling
        state bf_choice <<choice>>
        backfilling --> bf_choice: backfill message
update backfill_offset
        bf_choice --> [*]:  if backfill_offset >= target_offset
close backfill stream
        bf_choice --> backfilling : if backfill_offset < target_offset

        backfilling --> bf_choice: upstream message
target_offset #colone; upstream_offset
    }
    Backfilling --> SourceCatchingUp
    state SourceCatchingUp {
        [*] --> source_catching_up
        state choice <<choice>>
        source_catching_up --> source_catching_up: backfill message
ignore
        source_catching_up --> choice: upstream message
        choice --> source_catching_up:   if upstream_offset < backfill_offset

        choice --> [*]:   if upstream_offset >= backfill_offset

    }
    SourceCatchingUp --> Finished
    Finished --> ForwardingUpstream

Connector APIs
I investigated some connectors' API docs to see whether they can get latest_offset, and has_message_available
Note: it might be tempting to just use poll to check has_message_available, but not all connector APIs will return sth like NoMoreMessage in poll. We cannot distinguish timeout from no more messages in this case.

Kafka

latest_offset: high_watermark-1
has_message_available: high_watermark>low_watermark

Kinesis

Does not support neither latest_offset nor has_message_available directly.

DescribeStream return EndingSequenceNumber only for closed shards (after shard merge/split), not open ones.

However, GetRecords will return MillisBehindLatest.  "A value of zero indicates that record processing is caught up, and there are no new records to process at this moment." So we can use this API to workaround (we will fetch and drop data). Note that

Pulsar

supports getLastMessageIds
supports hasMessageAvailable. It explicitly mentioned that "This check can be used by an application to scan through a topic and stop when the reader reaches the current last published message.".

Note: another possibly useful ability is "fetch last 1/N messages". But this is subtle. We need to make sure the message is "inclusive".

Kafka: supports OffsetTail(n) natively (will become -n in the protocol). And Kafka offset is consecutive.
Pulsar: supports startMessageId(MessageId.latest).startMessageIdInclusive() to read last 1. But not last N.
Kinesis: seems not supported.

We've implemented 1-3 for kafka with:
/// Information used to determine whether we should start and finish source backfill.
///
/// XXX: if a connector cannot provide the latest offsets (but we want to make it shareable),
/// perhaps we should ban blocking DDL for it.
#[derive(Debug, Clone)]
pub enum BackfillInfo {
    HasDataToBackfill {
        /// The last available offsets for each split (**inclusive**).
        ///
        /// This will be used to determine whether source backfill is finished when
        /// there are no _new_ messages coming from upstream `SourceExecutor`. Otherwise,
        /// blocking DDL cannot finish until new messages come.
        ///
        /// When there are upstream messages, we will use the latest offsets from the upstream.
        latest_offset: String,
    },
    /// If there are no messages in the split at all, we don't need to start backfill.
    /// In this case, there will be no message from the backfill stream too.
    /// If we started backfill, we cannot finish it until new messages come.
    /// So we mark this a special case for optimization.
    NoDataToBackfill,
}


Modal Dialog box not displayed

Select one ...  (check one with "x")
[x] bug
[ ] feature request
[ ] enhancement

Expected behavior
Modal dialog box should open when button is clicked
Actual behavior
Modal does not open and the following error is shown in the web console:
ERROR Error: Found the synthetic listener @fadeDown.done. Please include either "BrowserAnimationsModule" or "NoopAnimationsModule" in your application.
Reproduction of behavior
Environment details

Angular version: 5.0.3

Clarity version: 0.10.15

OS and version: Windows 10

Browser: [Chrome XX | Firefox XX]

Clarity not support Angular 5 ?

I updated to 4/4/6

I'm using Clarity Design System's Modal and I only had this problem when adding storybook to my angular application, but the solution is similar.. just add the BrowserAnimationsModule to the moduleMetadata imports and it should work.
export const withDefault = () => ({
    component: ModalComponent,
    props: { ... },
    moduleMetadata: {
        imports: [ ..., BrowserAnimationsModule ],
    },
});


Allow direct transitions between stickyTop and stickyBottom modes

This commit adds checks to handleScroll() to allow for direct transitions between the stickyTop and stickyBottom modes.
This fixes the problem reported in issue #24, where a programmatic scroll (window.scrollTo()) issues a single event, that event causes a state change (say from stickyTop to relative), but does not actually reposition the StickyBox where it should be.

Fantastic work! I also realised that my trouble shooting advice of

window.scrollTo doesn't fire a scroll event. So you need to do it manually`

is a lie!
It just didn't work as expected because of the bug you've exposed (and fixed!) and adding a window.dispatchEvent(new Event('scroll')); was just a hack to make the scroll handler run twice.
I also appreciate that you started cleaning up that scroll handler a little.


point cloud data

What feature or requirment is this related to? Please describe
This code example relates to the feature or requirement mentioned on #<github_issue_here>
!pip install mapillary
import mapillary as mly
mly.interface.set_access_token('MLY|')
response = mly.interface.image_from_key(id,fields = ['sfm_cluster'])
sfm_cluster_url =
https:\\/\\/scontent-iad3-1.xx.fbcdn.net\\/m1\\/v\\/t0.40383-6\\/An8Lx-3QlYDKv2YxzpOgOEJeMdx1v4mm8Xvczm4oxncKAIlA_cmDpxwXsy65cnsTUmM6mgq9dpooROG658tIhEyOIrA8uHcqeFK5gB9TMikR9788Ku_awoEbtFHGJg2X2jPKjy0padhn0w?ccb=10-5&oh=00_AT_Tar_zzekzP0jDvbYAeHCNxU7GITO5m5HtEYy95wwm2w&oe=632180A9&_nc_sid=0f319a

real_url = 'https:\\scontent-iad3-1.xx.fbcdn.net/m1/v/t0.40383-6/An8Lx-3QlYDKv2YxzpOgOEJeMdx1v4mm8Xvczm4oxncKAIlA_cmDpxwXsy65cnsTUmM6mgq9dpooROG658tIhEyOIrA8uHcqeFK5gB9TMikR9788Ku_awoEbtFHGJg2X2jPKjy0padhn0w?ccb=10-5&oh=00_AT_Tar_zzekzP0jDvbYAeHCNxU7GITO5m5HtEYy95wwm2w&oe=632180A9&_nc_sid=0f319a'

I want to get the point of cloud of each image so I can reconstruction the camera position  and get better visualization, positioning, however. When I try to get the url that links to sfm_cluster, it returns to me a binary stream. After that , I write it content into a txt file,
but it becomes messy code. Is there any way to decode the messy code or api that I can get the *.ply format file?
Thank you!
Additional context

Yeah, right now, there isn't any mechanism to decode the contents from the URL that sfm_cluster returns. @cbeddow would you know of a way to do this?

Yeah, right now, there isn't any mechanism in the SDK to decode the contents from the URL that sfm_cluster returns. @cbeddow would you know of a way to do this?

So, it means that there is no solution to decode the contents of this url? I wonder if I could know it encoding type, then Maybe I can try to decode myself?

There is a way to decode the contents. I meant to say that this solution has not yet been integrated into the SDK.
The sfm_cluster property returns a link that points to an encoded .ply format file.
Can you please send me the file you have right now? I can look into this and integrate a solution to decode the SFM as well.

@neverland-haha, there is a way to decode the contents. I meant to say that this solution has not yet been integrated into the SDK.
The sfm_cluster property returns a link that points to an encoded .ply format file.
Can you please share the file here that you have right now? I can look into this and integrate a solution to decode the SFM as well.

This this the  url tha sfm_cluster  returns.
https:\\scontent-iad3-1.xx.fbcdn.net/m1/v/t0.40383-6/An8Lx-3QlYDKv2YxzpOgOEJeMdx1v4mm8Xvczm4oxncKAIlA_cmDpxwXsy65cnsTUmM6mgq9dpooROG658tIhEyOIrA8uHcqeFK5gB9TMikR9788Ku_awoEbtFHGJg2X2jPKjy0padhn0w?ccb=10-5&oh=00_AT_Tar_zzekzP0jDvbYAeHCNxU7GITO5m5HtEYy95wwm2w&oe=632180A9&_nc_sid=0f319a

@neverland-haha, I was able to write a quick python script to solve this issue.
import requests
import json
import zlib

def main():
    # change URL here as per need
    sfm_content_url = 'https://scontent-iad3-1.xx.fbcdn.net/m1/v/t0.40383-6/An8Lx-3QlYDKv2YxzpOgOEJeMdx1v4mm8Xvczm4oxncKAIlA_cmDpxwXsy65cnsTUmM6mgq9dpooROG658tIhEyOIrA8uHcqeFK5gB9TMikR9788Ku_awoEbtFHGJg2X2jPKjy0padhn0w?ccb=10-5&oh=00_AT_Tar_zzekzP0jDvbYAeHCNxU7GITO5m5HtEYy95wwm2w&oe=632180A9&_nc_sid=0f319a'

    # decompress sfm content, and return dict
    print(decompress_sfm_content(sfm_content_url))

def decompress_sfm_content(sfm_content_url: str) -> dict:
    # make get request with python and get content

    response = requests.get(sfm_content_url)
    content = response.content

    # decompress zlib content
    return json.loads(zlib.decompress(content).decode('utf-8'))

if __name__ == '__main__':
    main()

I'll be adding this function soon in the SDK. Please let me know if this resolves your issue.
And thanks for opening this issue! I learned something new today.

@neverland-haha, I was able to write a quick python script for this.
import requests
import json
import zlib

def main():
    # change URL here as per need
    sfm_content_url = 'https://scontent-iad3-1.xx.fbcdn.net/m1/v/t0.40383-6/An8Lx-3QlYDKv2YxzpOgOEJeMdx1v4mm8Xvczm4oxncKAIlA_cmDpxwXsy65cnsTUmM6mgq9dpooROG658tIhEyOIrA8uHcqeFK5gB9TMikR9788Ku_awoEbtFHGJg2X2jPKjy0padhn0w?ccb=10-5&oh=00_AT_Tar_zzekzP0jDvbYAeHCNxU7GITO5m5HtEYy95wwm2w&oe=632180A9&_nc_sid=0f319a'

    # decompress sfm content, and return dict
    print(decompress_sfm_content(sfm_content_url))

def decompress_sfm_content(sfm_content_url: str) -> dict:
    # make get request with python and get content

    response = requests.get(sfm_content_url)
    content = response.content

    # decompress zlib content
    return json.loads(zlib.decompress(content).decode('utf-8'))

if __name__ == '__main__':
    main()

I'll be adding this function soon in the SDK. Please let me know if this resolves your concern.
And thanks for opening this issue! I learned something new today.

It does solve my problem,  Thank you very much!

@neverland-haha, I'll close when I finally merge the PR for this function in the SDK. Reopening for now.

@neverland-haha, I'll close when I finally merge the PR for this function in the SDK. Reopening for now.

import mapillary as mly
import requests
import requests
import json
import zlib

mly.interface.set_access_token('MLY|5175886149161658|4099af83a040b4c64a3891bdcc8314cf')
data = mly.interface.image_from_key('1239407999827329',fields = ['sfm_cluster'])
result = eval(data)
url = result['features']['properties']['sfm_cluster']['url'].replace('\\','')
response = requests.get(url,stream=True,timeout=10)
info = json.loads(zlib.decompress(response.content).decode('utf-8'))
str_info = str(info)
length = len(str_info)
real_info = str_info[1:length-1]
dict_info = eval(real_info)
print(dict_info)

The dict_info contains too much infomation, but I want to retrive the earing angle and the pitch angle of the camera from the point cloud data. How can I know that point that I want?The mly.interface.image_from_key function returns me the id of point of cloud. But I can't find it in dict_info.

Hi, @neverland-haha! I haven't yet had the time to explore this, but some intuition that I can help with maybe.
After storing the data as a JSON, I found there are 4 keys in the dictionary.

The shots and points are the most heavy data keys.
I'm not sure myself what to do for the earing angle or the pitch angle in this case (I still lack some more advanced GIS knowledge), but I found this field that may (?) help you,

@cbeddow what would you suggest to do here in this case?

@Rubix982 @neverland-haha in the latest JSON example above, if you need the camera bearing angle, you will work with the rotation of the image: https://opensfm.org/docs/cam_coord_system.html
It should be the Z-Axis is telling you which angle the camera is facing. I am not sure if that is a 0 to 360 value though. The full data there you see is designed to work with OpenSfM library in Python so I suggest to go from Mapillary Python SDK to OpenSfM to work with the results you get from the JSON: https://opensfm.org/docs/dataset.html#reconstruction-file-format

Hi, @neverland-haha! I haven't yet had the time to explore this, but some intuition that I can help with maybe.
After storing the data as a JSON, I found there are 4 keys in the dictionary.

The shots and points are the most heavy data keys.
I'm not sure myself what to do for the earing angle or the pitch angle in this case (I still lack some more advanced GIS knowledge), but I found this field that may (?) help you,

@cbeddow what would you suggest to do here in this case?

Thanks. I' ll try and give you the answer. But As the doc says: Y-Axis is telling the angle the camera is up or down, however, when I change the units from rad to degree,some angle even greater than 90°. I'll check the process again.

@Rubix982 @neverland-haha in the latest JSON example above, if you need the camera bearing angle, you will work with the rotation of the image: https://opensfm.org/docs/cam_coord_system.html
It should be the Z-Axis is telling you which angle the camera is facing. I am not sure if that is a 0 to 360 value though. The full data there you see is designed to work with OpenSfM library in Python so I suggest to go from Mapillary Python SDK to OpenSfM to work with the results you get from the JSON: https://opensfm.org/docs/dataset.html#reconstruction-file-format

Thanks, I' ll try and give you the answer. But As the doc says: Y-Axis is telling the angle the camera is up or down, however, when I change the units from rad to degree,some angle even greater than 90°. I'll check the process again.


Fixed navbar with content scrolling

Navbar is now fixed, and content scrolls under it. This looks much better :3
This means that it fixes bungee scrolling, so Apple devices aren’t so retarded.

I liked my border radius, it looked nice on mobile. but I'll pull it in cause i like you

Lel it wasn't ready xD

Awe snap my bad :')

Ill just continue this later on. I have to work on categories right now.  Writing up some papers for plans.

Alright cool thank you!


Change to respect the Django widget format

Thanks @rgs258 - we must've never hit this edge case in OTIS.


add UMD to allow module exporting

Hi CrossLead team! Thank you for making this package available!  My name is Ted, I came across your package while trying to find a way to incorporate the Zendesk embeddable into our company's application in a sane, testable way (easier said than done).
We use npm + browserify for our dependency management, however, so if we wanted to use angular-zendesk-widget as-is we would need to add some sort of debowerify step in our build, which I'd rather avoid.
Did you have any existing plans to offer this package via npm as well?  If not, I believe that with a couple more commands to actually publish, this PR would enable you to offer NPM support by adding a UMD definition to the concat step of your build.  (https://github.com/umdjs/umd).
I'm not sure what your procedures are for accepting PRs, so I welcome feedback and am happy to work with you to make sure I've adhered to your requirements.
And again, thanks for making this!

Thanks @tedwardmah! Although we haven't used UMD patterns, your changes look good from a quick glance. Most likely it'll take until the weekend before I'll have time to review more closely, merge, and publish new versions (both on npm and bower).

@yangchristian thanks for the quick reply, take your time and let me know what you think

@tedwardmah: Just published v0.1.3 to both npm and bower. Thanks again!

hey @yangchristian thanks again for your quick work on this.  I wasn't able to work on integrating until this week, but I finally got to close out the ticket and happy to report that I'm pointing to the version on NPM currently.  Thanks!


getAdvertisingPacketBetween() in AdvertisingPacketUtil returns incorrect list

In the line a) the index of the first incorrect packet will be found, while in line b) the index of the last correct packet will be found. The problem is, that we did not adjust line b) but line c) resulting in adjusting both cases, a) and b).
// find the index of the last advertising packet with a timestamp
// smaller than the specified endTimestamp
int endIndex = advertisingPackets.size() - 1;
if (endTimestamp < latestAdvertisingPacket.getTimestamp()) {
    // figure out if the end timestamp is before or after the midst advertising packet
    ListIterator<P> listIterator;
    if (endTimestamp < midstAdvertisingPacket.getTimestamp()) {
        // end timestamp is in the first half of advertising packets
        // start iterating from the beginning
        listIterator = advertisingPackets.listIterator(startIndex);
        while (listIterator.hasNext()) {
            if (listIterator.next().getTimestamp() >= endTimestamp) {
                endIndex = listIterator.previousIndex();   // ---> a)
                break;
            }
        }
    } else {
        // end timestamp is in the second half of advertising packets
        // start iterating from the end
        listIterator = advertisingPackets.listIterator(advertisingPackets.size());
        while (listIterator.hasPrevious()) {
            if (listIterator.previous().getTimestamp() < endTimestamp) {
                endIndex = listIterator.nextIndex() + 1;   // ---> b)
                break;
            }
        }
    }
}

return new ArrayList<>(advertisingPackets.subList(startIndex, endIndex + 1));   // ---> c)

Any updates from this?

Please excuse the delay, we're on a tight schedule this week and won't be able to work on this repo before the 22nd.
The real issue here is that the method in question is lacking unit tests to assert that it behaves as expected. We'll add tests as soon as time permits.

Thank you


Geocode users

@elight @lenary Should we close this?
I'm back to using my personal github more and this is forever on my "mentions"


[Feature] SAC RGB(D) Baseline and improved memory efficiency

Tests on new SAC state and SAC RGBD baselines.
black line is state based, green is 64x64 RGB images, purple is 128x128 RGB images. Note 128x128 (the default camera size in most tasks) runs very slowly due to very slow update times. Eventually will make a sac_fast version based on leanrl.


rtd container uses wait-for-it, ensuring postfix before starting supe…

…rvisord. fixes #5.

@rboyd, thanks for the pull!
I'll review and test in a week or two (on vacation).


Documentation: Enhancement Of Contributing.md

The contributing.md file is not in a structured manner. Also some sections are missing. I can enhance it in a well structured manner with proper details and some additional sections also that will make the contributors easier to contribute. Please can you assign me this task?
Wanted to do the following changes :-

Enhance the project structure
Add need help with the basics section
Add project structure section
Add additional method of contributing through GitHub desktop section
Add good coding practices section
Add the help and support section
Add the pull request section
Add issue report process section

Hey @milansamuel609 ! 👋
First off, I want to extend a huge thank you to all of you for your amazing contributions to this project! Your support and hard work mean a lot, and I truly appreciate everything you do. 🙏
I have a small favor to ask: if you think I’ve been helpful and supportive throughout your contributions, I would be honored if you could nominate me by filling out this form: Nominate me.
Your nomination would mean the world to me, and it’s a great way to help me continue supporting our awesome community. Thanks again for all the amazing work, and let's keep making great things happen! 🚀
Best,
Vishesh


should addEventListener throw on invalid requireCancelable value?

Is it legitimate for addEventListener to know whether each type of event is cancelable and be able to throw when asking for a cancelable event of a type that is never cancelable?  Perhaps that should just silently proceed with uncancelable events?

At minimum, Chrome would need to provide a devtools warning in this case.
Are there cases where throwing would be useful for browser feature detection? For instance, if an event type is cancelable in some browsers but not others, then if addEventListener throws, you can test if the event type is cancelable on the current browser, and if not, use some alternative behavior.

Are there cases where throwing would be useful for browser feature detection?

I think relying on exceptions for feature detection is generally discouraged.  I mentioned here that I could imagine us wanting to add an API like TouchEvent.supportsCancelation if we someday decide to revoke the ability for a poorly performing page to cancel any touch events.
Still, I agree the exception might be more useful than silently doing the wrong thing.  Silently doing the wrong thing is more "webby" though :-)

BTW I filed #8 for the general issue of feature detection for this API.


Error compiling properties with init value expressions

When compiling an existing project I get the error "Cannot interpret AST BinaryExpression" because some properties in a class have an init value which is an expression.
  properties: {
    minimum: {
      refine: true,
      init: Number.MAX_VALUE * -1
    },
    maximum: {
      refine: true,
      init: Number.MAX_VALUE
    },
    value: {
      refine: true,
      init: new Date().getMonth()+1
    }
  }

@sambu16 @johnspackman Is this still a problem or can the issue be closed?

It does still seem to be an issue even with the latest released compiler:
2019-05-01 05:28:37.729 Cannot interpret AST BinaryExpression at plaid.ui.form.MonthSpinner [32,34]
I have worked around this myself by either setting the values in constructor or changing them as appropriate, e.g. -Number.MAX_VALUE