HBASE-23194 : Remove unused methods from TokenUtil

Jira: https://issues.apache.org/jira/browse/HBASE-23194

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

:blue_heart:
reexec
1m 31s
Docker mode activated.

_ Prechecks _

:green_heart:
dupname
0m  0s
No case conflicting files found.

:green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

:green_heart:
@author
0m  0s
The patch does not contain any @author tags.

:yellow_heart:
test4tests
0m  0s
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ master Compile Tests _

:green_heart:
mvninstall
6m 13s
master passed

:green_heart:
compile
0m 55s
master passed

:green_heart:
checkstyle
1m 30s
master passed

:green_heart:
shadedjars
5m  0s
branch has no errors when building our shaded downstream artifacts.

:green_heart:
javadoc
0m 36s
master passed

:blue_heart:
spotbugs
4m 30s
Used deprecated FindBugs config; considering switching to SpotBugs.

:green_heart:
findbugs
4m 28s
master passed

_ Patch Compile Tests _

:green_heart:
mvninstall
5m 21s
the patch passed

:green_heart:
compile
0m 56s
the patch passed

:green_heart:
javac
0m 56s
the patch passed

:green_heart:
checkstyle
1m 27s
the patch passed

:green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

:green_heart:
shadedjars
4m 58s
patch has no errors when building our shaded downstream artifacts.

:green_heart:
hadoopcheck
17m  2s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

:green_heart:
javadoc
0m 37s
the patch passed

:green_heart:
findbugs
4m 38s
the patch passed

_ Other Tests _

:green_heart:
unit
151m 38s
hbase-server in the patch passed.

:green_heart:
asflicense
0m 28s
The patch does not generate ASF License warnings.

214m  2s

Subsystem
Report/Notes

Docker
Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/1/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/737

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux e7e72920a091 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-737/out/precommit/personality/provided.sh

git revision
master / 4d414020bb

Default Java
1.8.0_181

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/1/testReport/

Max. process+thread count
4427 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/1/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

:blue_heart:
reexec
1m 13s
Docker mode activated.

_ Prechecks _

:green_heart:
dupname
0m  0s
No case conflicting files found.

:green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

:green_heart:
@author
0m  0s
The patch does not contain any @author tags.

:yellow_heart:
test4tests
0m  0s
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ master Compile Tests _

:green_heart:
mvninstall
5m 54s
master passed

:green_heart:
compile
0m 58s
master passed

:green_heart:
checkstyle
1m 28s
master passed

:green_heart:
shadedjars
4m 58s
branch has no errors when building our shaded downstream artifacts.

:green_heart:
javadoc
0m 40s
master passed

:blue_heart:
spotbugs
4m 39s
Used deprecated FindBugs config; considering switching to SpotBugs.

:green_heart:
findbugs
4m 37s
master passed

_ Patch Compile Tests _

:green_heart:
mvninstall
6m 47s
the patch passed

:green_heart:
compile
1m  8s
the patch passed

:green_heart:
javac
1m  8s
the patch passed

:green_heart:
checkstyle
1m 34s
the patch passed

:green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

:green_heart:
shadedjars
5m 22s
patch has no errors when building our shaded downstream artifacts.

:green_heart:
hadoopcheck
19m 31s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

:green_heart:
javadoc
0m 36s
the patch passed

:green_heart:
findbugs
4m 38s
the patch passed

_ Other Tests _

:broken_heart:
unit
229m 24s
hbase-server in the patch failed.

:green_heart:
asflicense
0m 26s
The patch does not generate ASF License warnings.

296m 43s

Subsystem
Report/Notes

Docker
Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/2/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/737

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 564e463d48e0 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-737/out/precommit/personality/provided.sh

git revision
master / da9a53ed4d

Default Java
1.8.0_181

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/2/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/2/testReport/

Max. process+thread count
4593 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/2/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

Please review @petersomogyi @saintstack

:broken_heart: -1 overall

Vote
Subsystem
Runtime
Comment

:blue_heart:
reexec
0m 31s
Docker mode activated.

_ Prechecks _

:green_heart:
dupname
0m  0s
No case conflicting files found.

:green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

:green_heart:
@author
0m  0s
The patch does not contain any @author tags.

:yellow_heart:
test4tests
0m  0s
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ master Compile Tests _

:broken_heart:
mvninstall
6m 30s
root in master failed.

:green_heart:
compile
0m 57s
master passed

:green_heart:
checkstyle
1m 29s
master passed

:green_heart:
shadedjars
5m  1s
branch has no errors when building our shaded downstream artifacts.

:green_heart:
javadoc
0m 36s
master passed

:blue_heart:
spotbugs
4m 28s
Used deprecated FindBugs config; considering switching to SpotBugs.

:green_heart:
findbugs
4m 26s
master passed

_ Patch Compile Tests _

:green_heart:
mvninstall
5m 37s
the patch passed

:green_heart:
compile
0m 56s
the patch passed

:green_heart:
javac
0m 56s
the patch passed

:broken_heart:
checkstyle
1m 28s
hbase-server: The patch generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1)

:green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

:green_heart:
shadedjars
4m 55s
patch has no errors when building our shaded downstream artifacts.

:green_heart:
hadoopcheck
17m  2s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

:green_heart:
javadoc
0m 35s
the patch passed

:green_heart:
findbugs
4m 34s
the patch passed

_ Other Tests _

:broken_heart:
unit
166m 47s
hbase-server in the patch failed.

:green_heart:
asflicense
0m 28s
The patch does not generate ASF License warnings.

228m 26s

Subsystem
Report/Notes

Docker
Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/3/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/737

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 80542ffb968a 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-737/out/precommit/personality/provided.sh

git revision
master / 42d535a57a

Default Java
1.8.0_181

mvninstall
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/3/artifact/out/branch-mvninstall-root.txt

checkstyle
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/3/artifact/out/diff-checkstyle-hbase-server.txt

unit
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/3/artifact/out/patch-unit-hbase-server.txt

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/3/testReport/

Max. process+thread count
4464 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/3/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.

:confetti_ball: +1 overall

Vote
Subsystem
Runtime
Comment

:blue_heart:
reexec
0m 33s
Docker mode activated.

_ Prechecks _

:green_heart:
dupname
0m  0s
No case conflicting files found.

:green_heart:
hbaseanti
0m  0s
Patch does not have any anti-patterns.

:green_heart:
@author
0m  0s
The patch does not contain any @author tags.

:yellow_heart:
test4tests
0m  0s
The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.

_ master Compile Tests _

:green_heart:
mvninstall
6m 49s
master passed

:green_heart:
compile
1m  6s
master passed

:green_heart:
checkstyle
1m 42s
master passed

:green_heart:
shadedjars
5m 33s
branch has no errors when building our shaded downstream artifacts.

:green_heart:
javadoc
0m 38s
master passed

:blue_heart:
spotbugs
4m 32s
Used deprecated FindBugs config; considering switching to SpotBugs.

:green_heart:
findbugs
4m 29s
master passed

_ Patch Compile Tests _

:green_heart:
mvninstall
5m 27s
the patch passed

:green_heart:
compile
0m 58s
the patch passed

:green_heart:
javac
0m 58s
the patch passed

:green_heart:
checkstyle
1m 28s
the patch passed

:green_heart:
whitespace
0m  0s
The patch has no whitespace issues.

:green_heart:
shadedjars
5m  0s
patch has no errors when building our shaded downstream artifacts.

:green_heart:
hadoopcheck
17m  0s
Patch does not cause any errors with Hadoop 2.8.5 2.9.2 or 3.1.2.

:green_heart:
javadoc
0m 34s
the patch passed

:green_heart:
findbugs
4m 33s
the patch passed

_ Other Tests _

:green_heart:
unit
156m 33s
hbase-server in the patch passed.

:green_heart:
asflicense
0m 26s
The patch does not generate ASF License warnings.

219m 32s

Subsystem
Report/Notes

Docker
Client=19.03.4 Server=19.03.4 base: https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/4/artifact/out/Dockerfile

GITHUB PR
https://github.com/apache/hbase/pull/737

Optional Tests
dupname asflicense javac javadoc unit spotbugs findbugs shadedjars hadoopcheck hbaseanti checkstyle compile

uname
Linux 94aa602aad2d 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 GNU/Linux

Build tool
maven

Personality
/home/jenkins/jenkins-slave/workspace/HBase-PreCommit-GitHub-PR_PR-737/out/precommit/personality/provided.sh

git revision
master / 42d535a57a

Default Java
1.8.0_181

Test Results
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/4/testReport/

Max. process+thread count
4754 (vs. ulimit of 10000)

modules
C: hbase-server U: hbase-server

Console output
https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-737/4/console

versions
git=2.11.0 maven=2018-06-17T18:33:14Z) findbugs=3.1.11

Powered by
Apache Yetus 0.11.0 https://yetus.apache.org

This message was automatically generated.


Bug in Calculator code

Describe the bug

When we insert the operators many times still it calculates the correct value.

Possible solution

There should some error value like  Syntax error.

Screenshots

I would like to work on this issue as a part of GSSoC'21

But if it's working well, then we should not show such messages

But a real calculator would show an error message. This is not the right math

@Karthik-Nayak98 Many calculators work differently, and ours is no exception, so we can't go ahead with this issue right now. Hope you understand!
But thanks for opening this issue! I am closing this as of now :)


Add bsides to events/meetups

Add https://www.bsidesedmonton.org/ to events and meetups.
image:
https://ik.imagekit.io/des/events/bsides.png

Please assign this to me

I've already made my contribution for Hacktoberfest, which starts on Monday.

PRs do not count for hacktoberfest until after october 1st


Add maven-properties class, used by common-api

This class was removed in commit 2f3642f564d39692f174b4c46fcc95ca371becae.
Signed-off-by: Tobias Olausson<EMAIL_ADDRESS>

My bad hides away in shame! Thanks @tobsan !


Refactor dashboard/feed.tmpl

Refactor ActionType to models/activities/action_type.go
Replace the magic number in feed.tmlp with InAction

Dashboard fails to load for me with 500 on my local instance after this:
Render failed, failed to render template: user/dashboard/dashboard, error: template error: builtin(static):user/dashboard/feeds:14:20 : executing "user/dashboard/feeds" at <.GetOpType.InActions>: can't evaluate field InActions in type activities.ActionType
----------------------------------------------------------------------
					{{if .GetOpType.InActions "create_repo"}}
					               ^

Let me check It.

Yes, I'm at https://github.com/go-gitea/gitea/commit/bafe1334c6e46c0a0398c645da97b53f42996ab8.

Yes, I'm at bafe133.

Thanks, indeed this function was missing.

Must be something else. I defintely got https://github.com/go-gitea/gitea/commit/30cea70dd8e0a4583d8f63795b21551ce44ce223 on my main branch that exactly matches the remote.

https://github.com/go-gitea/gitea/blob/bafe1334c6e46c0a0398c645da97b53f42996ab8/models/activities/action.go#L129-L136
Does this code exist locally? It's kind of weird.

Yes, it's there. Maybe I will dive deeper later, unless you find it first. You should be able to reproduce this hopefully. Or maybe it's something weird about my database.

It can't be related to database.
Have you re-compiled and restarted the Gitea instance? Or could you try to do it manually in case it's the air bug?

I ran make clean-all build and that seems to have fixed it. Bad cache somewhere I assume.


Fix tracing in postsubmit tests

Please provide a description of this PR:
Missed a PULL_NUMBER
However the URL will be incorrect. Looking into that some more.

@ericvn: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name
Commit
Details
Required
Rerun command

release-notes_istio
0b590fb0966bce0cd437349010ed3a475510306b
link
true
/test release-notes

Instructions for interacting with me using PR comments are available here.  If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.


[FEA] Add tests for configs in BatchScan Readers

Is your feature request related to a problem? Please describe.
As part of audit some of the configs in parquet and orc seem stood out and we may want to test that they work on the GPU. Needs more triaging.
hadoopConf.set(ParquetInputFormat.READ_SUPPORT_CLASS, classOf[ParquetReadSupport].getName)
    hadoopConf.set(
      ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA,
      readDataSchemaAsJson)
    hadoopConf.set(
      ParquetWriteSupport.SPARK_ROW_SCHEMA,
      readDataSchemaAsJson)
    hadoopConf.set(
      SQLConf.SESSION_LOCAL_TIMEZONE.key,
      sparkSession.sessionState.conf.sessionLocalTimeZone)
    hadoopConf.setBoolean(
      SQLConf.NESTED_SCHEMA_PRUNING_ENABLED.key,
      sparkSession.sessionState.conf.nestedSchemaPruningEnabled)

...
hadoopConf.setBoolean(
      SQLConf.PARQUET_BINARY_AS_STRING.key,
      sparkSession.sessionState.conf.isParquetBinaryAsString)

Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context, code examples, or references to existing implementations about the feature request here.

ParquetReadSupport.SPARK_ROW_REQUESTED_SCHEMA and ParquetWriteSupport.SPARK_ROW_SCHEMA are part of a hacked up way to pass the row schema from one part of spark to another.  We can ignore them.
SQLConf.SESSION_LOCAL_TIMEZONE is a hacked up way to pass the time zone to another part of Spark when reading the data.  When we support any timezone besides UTC then we can look at supporting something like this.
SQLConf.NESTED_SCHEMA_PRUNING_ENABLED is for

Prune nested fields from a logical relation's output which are unnecessary in
satisfying a query. This optimization allows columnar file format readers to avoid
reading unnecessary nested column data. Currently Parquet and ORC are the
data sources that implement this optimization.

When we support reading structs we should look into this config.  Until then we can ignore it.
SQLConf.PARQUET_BINARY_AS_STRING is for

Some other Parquet-producing systems, in particular Impala and older versions of
Spark SQL, do not differentiate between binary data and strings when writing out the
Parquet schema. This flag tells Spark SQL to interpret binary data as a string to provide
compatibility with these systems.

For now we don't support binary data, so we should verify that cudf converts binary data that it finds to a string when it is returned.
When we do support binary data, then we will need cudf to some how support this too.
I propose that we file follow on issues for schema pruning and binary as string as place holders for when we support those features.  For this issue we verify existing functionality for binary as string.  If we cannot support it then we need to fall back to the CPU if this config is set and we have a string column as input in the schema.

A little bit more information about this.  Spark stores the schema in the parquet output so setting PARQUET_BINARY_AS_STRING  has no impact on files produced by Spark.

I am currently stuck on getting a parquet file that can reproduce this issue.

So I read through the parquet reader code that we currently have in cudf and it is totally ignoring the UTF8 logical type and is only going off of the physical type for a string BYTE_ARRAY and FIXED_LEN_BYTE_ARRAY.  But I suspect that this is going to change in the future once we get support for reading Lists/arrays in.
Because I have been unable to find/make a parquet file that I can use to test this feature I am inclined to just fall back to the CPU in all cases where it is set, even though it should work with the current code.


Open files with issues in an editor

This PR fixes #38.
When xo is invoked with --open option, it automatically opens files with issues in a user's preferred editor (by checking $EDITOR environment variable):
$ xo --open

If $EDITOR is empty:

@sindresorhus PR updated according to suggestions.

Perfect! This is so useful :)

@jamestalmage could you provide a working command-line example using file:line:column


Add variable mesh 60-3km for forecast (outer)

Description
This PR adds specification for the variable mesh 60-3km. At this moment, this mesh was only tested for the forecast not the data assimilation. The namelist is made more configurable, adding high flexibility for the physical parameterizations selection. The tables for the Thompson microphysics are added. ForecastFromGFSAnalysesVariableMesh scenario is added for this functionality.
Issue closed
None
Tests completed
Tier 1:

[x] 3dvar_OIE120km_WarmStart
[ ] 3denvar_OIE120km_IAU_WarmStart
[x] 3dvar_OIE120km_ColdStart
[ ] 3dvar_O30kmIE60km_ColdStart
[x] 3denvar_O30kmIE60km_WarmStart
[x] eda_OIE120km_WarmStart
[x] getkf_OIE120km_WarmStart
[x] 4dhybrid_OIE120km_WarmStart
[ ] ForecastFromGFSAnalysesMPT

Tier 3 (optional):

[x] ForecastFromGFSAnalysesVariableMesh

If we want to run variable mesh experiments, we need to add the section 'staticstream' and define 'filePrefix' under it.

If we want to run variable mesh experiments, we need to add the section 'staticstream' and define 'filePrefix' under it. I am still trying it and let you know the update.

You mean for DA experiments with variable mesh?

If we want to run variable mesh experiments, we need to add the section 'staticstream' and define 'filePrefix' under it. I am still trying it and let you know the update.

You mean for DA experiments with variable mesh?

Yes! And we should be careful with its prefix "x1". For 60-3 km mesh, its prefix is x20.

If we want to run variable mesh experiments, we need to add the section 'staticstream' and define 'filePrefix' under it. I am still trying it and let you know the update.

You mean for DA experiments with variable mesh?

Yes! And we should be careful with its prefix "x1". For 60-3 km mesh, its prefix is x20.

I get that, but the goal of this PR is to make the variable mesh work for the forecast. Do you mind opening a PR after this to include changes for DA?
Regarding the prefix, for simplicity I changed the name of the mesh, graph, and static files to be x1, then I did not need to add another variable for that. Nonetheless, I agree the most correct thing is to keep it as x20. Will add that.

If we want to run variable mesh experiments, we need to add the section 'staticstream' and define 'filePrefix' under it. I am still trying it and let you know the update.

You mean for DA experiments with variable mesh?

Yes! And we should be careful with its prefix "x1". For 60-3 km mesh, its prefix is x20.

I get that, but the goal of this PR is to make the variable mesh work for the forecast. Do you mind opening a PR after this to include changes for DA?
Regarding the prefix, for simplicity I changed the name of the mesh, graph, and static files to be x1, then I did not need to add another variable for that. Nonetheless, I agree the most correct thing is to keep it as x20. Will add that.

I see. If this PR doesn't handle DA, I will submit a new one for DA cycling. Thanks!

UngribExt*    --    1   1  235gb 00:05 Q   --
ObsToIODA*    --    1   1  125gb 00:15 Q   --
I suggest using 128 cores per node for those jobs on Derecho to reduce the queue time.

UngribExt* -- 1 1 235gb 00:05 Q -- ObsToIODA* -- 1 1 125gb 00:15 Q --
I suggest using 128 cores per node for those jobs on Derecho to reduce the queue time.

I made those changes in latest commits. Thanks for testing it @mos3r3n!


Portfolio Modelling: Add proper portfolio model from "Portfolio Theory"

Most portfolio theories interpret the model of a portfolio as a sequence of identifiers with a number associated with them, from markowtiz in classical portfolio theory, to modern portfolio theory where negative numbers were allowed to symbolize short positions.
Stochastic portfolio theory models them with a random variable and Model-free portfolio theory with rough path trajectories.
The later 2 should be implemented in a different ticket


Add more dependabot configuration

Codecov Report

Merging #666 (22c7990) into main (41754b5) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main     #666   +/-   ##
=======================================
  Coverage   85.95%   85.95%
=======================================
  Files         312      312
  Lines       28974    28974
=======================================
  Hits        24906    24906
  Misses       4068     4068

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 41754b5...22c7990. Read the comment docs.


Feature request: Stop updating prices at midnight

Please note that this is about the prices attribute. Not prices_today and prices_tomorrow
I wonder if it is possible to stop updating the prices attribute at midnight. I see no good reason for deleting half of the data at that time. The reason it makes things difficult is that I want to have the cheapest hours between 18:00 and 06:00 next day. But when prices updates at midnight and the previous day is deleted from the list, it is impossible to find the cheapest hours anymore. The history is gone and from midnight I can only find the cheapest hours from 00:00 to 06:00.
The template below works as expected from 18:00 to midnight, but at midnight the prices attribute changes and I get a new set of 6 cheapest hours. At the worst, I could have the 6 cheapest hours between 18:00 and 06:00 before midnight and get new 6 hours after midnight so that I get 12 hour in total.
I see no good reason to delete data from the prices attribute at midnight. A lot of automation happens at night, too, and changing this list when there is no changes in the data it contains, makes building templates a lot more difficult than it has to be. In fact, I dont know how to solve the problem in a template alone and has to use python scripting instead. I can handle that, but users  with no experience in programming will struggle with making python scripts.

platform: template
sensors:
cheapest_hours_between_18_and_06:
value_template: >-
{% set l=state_attr('sensor.current_electricity_market_price', 'prices')[18:30]|sort(attribute='price') %}
{{ (now() >= as_datetime(l[0].time) and now() <= as_datetime(l[0].time) + timedelta(hours = 1))
or (now() >= as_datetime(l[1].time) and now() <= as_datetime(l[1].time) + timedelta(hours = 1))
or (now() >= as_datetime(l[2].time) and now() <= as_datetime(l[2].time) + timedelta(hours = 1))
or (now() >= as_datetime(l[3].time) and now() <= as_datetime(l[3].time) + timedelta(hours = 1))
or (now() >= as_datetime(l[4].time) and now() <= as_datetime(l[4].time) + timedelta(hours = 1))
or (now() >= as_datetime(l[5].time) and now() <= as_datetime(l[5].time) + timedelta(hours = 1)) }}

There has to be some moment past prices are purged and the prices attribute changes. This is now done at midnight, where yesterdays prices get purged and tomorrows prices become todays prices. This seemed the most convenient. I get your point, so maybe we can do this in a smarter way. Keeping the past prices for some extra time might solve your problem.

Yes, 48 different sensors will be useless for me.
Regarding the update of the prices attribute, why not update it between 13:00 and 14:00 when new prices are available and only then?

It is not about the update moment (this happens every hour), but the moment previous prices get purged. When prices before 13:00 would be removed, graphs would only contain prices from 13:00 onward. This would not fix the issue, but would place it on another moment in the day. When you need the cheapest prices between 10:00 and 18:00 for example.

I think a prices attribute which always contain 48 elements and changes only when tomorrow prices are available, would be perfect. I cant see any good reason for deleting elements in this list at other times of the day. What is deleting data and shrink the list at other times going to solve?
Each day when tomorrows prices are available, prices[0:23] are set with the todays prices and prices[24:47] are set to tomorrows prices. Does not need to be more complex than that.
And it makes templates a lot easier to make when this list always have 48 elements and only updates a single time each day when there is a change in data from entso-e between 13:00 & 14:00

I seem to be facing this problem too, hopefully it's something that could be relatively easily solved at this relatively early stage.

What about having the prices attribute always contain up to 72 hours worth of data.
Meaning that every update at 00:00 would remove the data > 24h ago.
Then you could also add an yesterday attribute containing yesterday's prices split from the list you already retrieved for the prices attribute.
This would also allow users to more easily determine when to charge their battery.
Since price hops would become more easily identifiable.

What about having the prices attribute always contain up to 72 hours worth of data. Meaning that every update at 00:00 would remove the data > 24h ago. Then you could also add an yesterday attribute containing yesterday's prices split from the list you already retrieved for the prices attribute. This would also allow users to more easily determine when to charge their battery. Since price hops would become more easily identifiable.

It would be better, but it will still update this attribute at midnight when there really is no changes in the data it contains. And if I understand you correctly, this list will still be variable in length and between 48 and 72 elements. That makes using it in Home Assistant templates more difficult than it has to be as elements change at midnight. You have to make some logic that can handle the list before midnight and another logic for after midnight.

The whole point for me was to have a list that does not change at midnight. As it is currently implemented, it is equl to the prices_today list for 14 hours and it change at midnigth as both prices_today and prices_tomorrow does. So the prices list could either be deleted or changed. It is redundant in its current state :-(

Could there be an option to choose preferred method when setting it up? "Update list at midnight" and "update list when new prices become available"?

So the prices list could either be deleted or changed. It is redundant in its current state :-(

or only use that, and simply adjust the frontend. Agree on the many redundant attributes, for me though only using this entity is the temporary solution
(ofc, I also manually edited the attributes out except for this list)

What about having the prices attribute always contain up to 72 hours worth of data. Meaning that every update at 00:00 would remove the data > 24h ago. Then you could also add an yesterday attribute containing yesterday's prices split from the list you already retrieved for the prices attribute. This would also allow users to more easily determine when to charge their battery. Since price hops would become more easily identifiable.

It would be better, but it will still update this attribute at midnight when there really is no changes in the data it contains. And if I understand you correctly, this list will still be variable in length and between 48 and 72 elements. That makes using it in Home Assistant templates more difficult than it has to be as elements change at midnight. You have to make some logic that can handle the list before midnight and another logic for after midnight.

What about retrieving up to 72 hours of data and in the integration pick the newest 48H to be put into the prices attribute.
Then the prices attribute won't be just today and tommorow combined and it would update at 13:00 (when new prices become available) and always have 48 values to go through in templating etc.

this has been implemented!

this has been implemented!

It has? In which version? I am using 0.1.1 and it is not implementet there.

Saw now that you have started with releases and this was implemented in main and ready for the next release.

@SVH-Powel it is implemented in #53 (and some bugfixes in commits to main after that).
@JaccoR  has made alot of quality of life improvements like this but hasn't made a release for them yet.
I'm cheating a little bit by overwriting the custom_component files with the latest copy of main, but I'm also looking forward towards a new x.x.2 release :)

There were still some bugs in the main copy, so I was waiting to release 0.1.2. They are fixed now though, so 0.1.2 is coming shortly!

Thank you so much for releasing :)


The mode is a lie?

Or at least, the datasheet and output don't seem to match up. Is this how your (v117) devices behave as well?
I've got three (v118) AGS02MA here, that I wasn't able to get into UGM3 measurements on some custom ESP IDF code. Thought I was going mad so I tried an Arduino uno with your library and see the same results, not really changing modes apparently.
Looking at the datasheet, I'd expect the status value to change.

b0000 (decimal 0) for ready + PPB mode.
b0010 (decimal 2) for ready + UGM3 mode.

Using an adaptation of the library example code, I set up one that alternates modes every 10 measurements. Except that it doesn't? Here's some output I'm seeing with the status being 0 throughout. And no noticeable change in measured value either (other than going down as it warms up).
/home/beanow/duino/AGS02MA_PPB/AGS02MA_PPB.ino
AGS02MA_LIB_VERSION: 0.1.3

BEGIN:	1
VERSION:	118
VERS:	118
MODE:	1	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
PPB:	85	0	0
MODE:	1	1
UGM3:	85	0	0
UGM3:	85	0	0
UGM3:	85	0	0
UGM3:	85	0	0
UGM3:	85	0	0
UGM3:	85	0	0
UGM3:	84	0	0
UGM3:	84	0	0
UGM3:	84	0	0
UGM3:	84	0	0
MODE:	1	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
PPB:	84	0	0
MODE:	1	1
UGM3:	83	0	0
UGM3:	83	0	0
UGM3:	83	0	0

//
//    FILE: AGS02MA_PPB.ino
//  AUTHOR: Rob Tillaart
// VERSION: 0.1.0
// PURPOSE: test application
//    DATE: 2021-08-12
//     URL: https://github.com/RobTillaart/AGS02MA
//

#include "AGS02MA.h"

AGS02MA AGS(26);

uint32_t rounds = 0;

void setup()
{
  Serial.begin(115200);
  Serial.println(__FILE__);

  Wire.begin();

  Serial.print("AGS02MA_LIB_VERSION: ");
  Serial.println(AGS02MA_LIB_VERSION);
  Serial.println();

  bool b = AGS.begin();
  Serial.print("BEGIN:\t");
  Serial.println(b);

  Serial.print("VERSION:\t");
  Serial.println(AGS.getSensorVersion());

  // pre-heating improves measurement quality
  // can be skipped
//  Serial.println("\nWarming up (120 seconds = 24 dots)");
//  while (AGS.isHeated() == false)
//  {
//    delay(5000);
//    Serial.print(".");
//  }
//  Serial.println();

  uint8_t version = AGS.getSensorVersion();
  Serial.print("VERS:\t");
  Serial.println(version);
}

void loop()
{
  delay(3000);

  uint8_t kind = rounds % 20;

  // Switch mode every 10 and 20 rounds.
  bool b;
  if (kind == 0) {
    b = AGS.setPPBMode();
    uint8_t m = AGS.getMode();
    Serial.print("MODE:\t");
    Serial.print(b);
    Serial.print("\t");
    Serial.println(m);
  } else if (kind == 10) {
    b = AGS.setUGM3Mode();
    uint8_t m = AGS.getMode();
    Serial.print("MODE:\t");
    Serial.print(b);
    Serial.print("\t");
    Serial.println(m);
  }

  // Read PPB in first half of a 20-round cycle.
  if (kind < 10) {
    uint32_t value = AGS.readPPB();
    Serial.print("PPB:\t");
    Serial.print(value);
    Serial.print("\t");
    Serial.print(AGS.lastStatus(), HEX);
    Serial.print("\t");
    Serial.print(AGS.lastError(), HEX);
    Serial.println();
  } else {
    uint32_t value = AGS.readUGM3();
    Serial.print("UGM3:\t");
    Serial.print(value);
    Serial.print("\t");
    Serial.print(AGS.lastStatus(), HEX);
    Serial.print("\t");
    Serial.print(AGS.lastError(), HEX);
    Serial.println();
  }

  rounds++;
}

// -- END OF FILE --

Thanks! Would be great to compare and know if something is different about the 118 version or whether it's the same for 117.
https://github.com/RobTillaart/AGS02MA/blob/e42a5387da9aa182446708972d3450896c69a919/AGS02MA.cpp#L104-L111
Comparing those it does look like a date in the first bytes btw.
Mine reports 21, 2, 3, 118, CRC looking like 2021-feb-3.
Sourced at https://nl.aliexpress.com/item/1005003131194702.html
Though I can test sketches or ship one if needed for comparison.

YES, found the sensor rather quick.
Ran your sketch inhouse
19:01:41.644 -> AGS02MA_LIB_VERSION: 0.1.3
19:01:41.644 ->
19:01:41.644 -> BEGIN:	1
19:01:41.644 -> VERSION:	117
19:01:41.737 -> VERS:	117
19:01:44.783 -> MODE:	1	0
19:01:44.830 -> PPB:	856	11	0
19:01:47.876 -> PPB:	848	11	0
19:01:50.876 -> PPB:	838	11	0
19:01:53.922 -> PPB:	831	11	0
19:01:56.962 -> PPB:	824	11	0
19:01:59.997 -> PPB:	819	11	0
19:02:03.043 -> PPB:	814	10	0
19:02:06.089 -> PPB:	810	10	0
19:02:09.134 -> PPB:	803	10	0
19:02:12.134 -> PPB:	800	10	0
19:02:15.180 -> MODE:	1	1
19:02:15.227 -> UGM3:	793	13	0                 <<<<<<<<<<<<<< bit strange
19:02:18.270 -> UGM3:	1852	12	0
19:02:21.316 -> UGM3:	1827	12	0
19:02:24.315 -> UGM3:	1809	12	0
19:02:27.363 -> UGM3:	1793	12	0
19:02:30.426 -> UGM3:	1781	12	0
19:02:33.439 -> UGM3:	1786	12	0
19:02:36.496 -> UGM3:	1758	12	0
19:02:39.531 -> UGM3:	1777	12	0
19:02:42.553 -> UGM3:	1786	12	0
19:02:45.563 -> MODE:	1	0
19:02:45.608 -> PPB:	1802	11	0     <<<<<<<  switching artefact?
19:02:48.684 -> PPB:	783	10	0
19:02:51.683 -> PPB:	782	10	0
19:02:54.729 -> PPB:	773	10	0
19:02:57.769 -> PPB:	756	10	0
19:03:00.816 -> PPB:	748	10	0
19:03:03.853 -> PPB:	739	10	0
19:03:06.887 -> PPB:	731	10	0
19:03:09.940 -> PPB:	726	10	0
19:03:12.985 -> PPB:	717	10	0
19:03:15.985 -> MODE:	1	1
19:03:16.031 -> UGM3:	715	13	0     <<<<<<<<<<<<<<<<<<< idem
19:03:19.067 -> UGM3:	1625	12	0
19:03:22.113 -> UGM3:	1614	12	0
19:03:25.159 -> UGM3:	1621	12	0

Although there seems to be a switching artefact I got quite different values for PPB and UGM3.

switching
OK, 2 second delay might solve it (will try later)
Wiring
Arduino UNO - IDE 1.18.19 - no resistors - 10 cm wires.
Values
I let it run for some time as it seems to drop its value still (sensor came from my working room which is more dusty?)
Don't know how far it will drop bit your values seem very low to me.
19:19:59.289 -> MODE:	1	0
19:19:59.336 -> PPB:	961	11	0
19:20:02.407 -> PPB:	418	10	0
19:20:05.418 -> PPB:	418	10	0
19:20:08.463 -> PPB:	419	10	0
19:20:11.483 -> PPB:	419	10	0
19:20:14.557 -> PPB:	416	10	0
19:20:17.551 -> PPB:	417	10	0
19:20:20.622 -> PPB:	418	10	0
19:20:23.633 -> PPB:	417	10	0
19:20:26.705 -> PPB:	414	10	0
19:20:29.682 -> MODE:	1	1
19:20:29.762 -> UGM3:	417	13	0
19:20:32.771 -> UGM3:	952	12	0
19:20:35.816 -> UGM3:	950	12	0
19:20:38.869 -> UGM3:	950	12	0
19:20:41.925 -> UGM3:	954	12	0
19:20:44.956 -> UGM3:	961	12	0
19:20:47.961 -> UGM3:	959	12	0
19:20:51.030 -> UGM3:	957	12	0
19:20:54.036 -> UGM3:	952	12	0
19:20:57.096 -> UGM3:	950	12	0

(You're from Netherlands?  Me ZO Brabant)

Yep, Rotterdam here.

Could they have changed (or messed up) the protocol for v118? :thinking:
I get no mode switching and no status byte at all.

The value dropping can take a long time indeed. Much longer than the 120s. Datasheet does mention something about that.

Haven't tracked exactly, but did take hours not minutes to settle on 60 PPB.
Switching to another sensor that was powered off, you can see it drop from higher values.

/home/beanow/duino/AGS02MA_PPB/AGS02MA_PPB.ino
AGS02MA_LIB_VERSION: 0.1.3

BEGIN:	1
VERSION:	118
VERS:	118
MODE:	1	0
PPB:	2133	0	0
PPB:	1671	0	0
PPB:	1466	0	0
PPB:	1330	0	0
PPB:	1234	0	0
PPB:	1157	0	0
PPB:	1098	0	0
PPB:	1047	0	0
PPB:	1004	0	0
PPB:	960	0	0
MODE:	1	1
UGM3:	886	0	0
UGM3:	855	0	0
UGM3:	830	0	0
UGM3:	807	0	0
UGM3:	786	0	0
UGM3:	766	0	0
UGM3:	749	0	0
UGM3:	735	0	0
UGM3:	717	0	0
UGM3:	703	0	0
MODE:	1	0
PPB:	680	0	0
PPB:	667	0	0
PPB:	657	0	0
PPB:	648	0	0
PPB:	639	0	0
PPB:	627	0	0
PPB:	622	0	0
PPB:	614	0	0
PPB:	606	0	0
PPB:	602	0	0
MODE:	1	1
UGM3:	586	0	0
UGM3:	580	0	0
UGM3:	574	0	0
UGM3:	566	0	0
UGM3:	560	0	0
UGM3:	556	0	0
UGM3:	549	0	0
UGM3:	543	0	0
UGM3:	539	0	0
UGM3:	533	0	0

I'll try mess around with the wiring. And maybe external power, currently running off USB power to the UNOs. 5v and short wires onto a breadboard from there. Though I do assume this is handled with the internal pullup.

If you get readings like that the signal is good to perfect, No pull ups needed.
An excellent reader about pull ups - http://www.gammon.com.au/forum/?id=10896&reply=5#reply5
(there is plenty more info there)

It's at least responding to gasses though. Here's holding a lighter gas

MODE:	1	0
PPB:	125	0	0
PPB:	124	0	0
PPB:	124	0	0
PPB:	124	0	0
PPB:	124	0	0
PPB:	124	0	0
PPB:	123	0	0
PPB:	1376	0	0
PPB:	6278	0	0
PPB:	12794	0	0
MODE:	1	1
UGM3:	25712	0	0
UGM3:	14832	0	0
UGM3:	9008	0	0
UGM3:	5105	0	0
UGM3:	3157	0	0
UGM3:	2148	0	0
UGM3:	1534	0	0
UGM3:	1143	0	0
UGM3:	818	0	0
UGM3:	560	0	0
MODE:	1	0
PPB:	314	0	0
PPB:	250	0	0
PPB:	209	0	0
PPB:	1259	0	0
PPB:	3814	0	0
PPB:	5548	0	0
PPB:	4027	0	0
PPB:	1876	0	0
PPB:	902	0	0
PPB:	381	0	0
MODE:	1	1
UGM3:	165	0	0
UGM3:	2142	0	0
UGM3:	17367	0	0
UGM3:<PHONE_NUMBER>	0	0    <==== weird, overflow?
UGM3:	19033	0	0
UGM3:	11041	0	0
UGM3:	5453	0	0
UGM3:	3070	0	0
UGM3:	1946	0	0
UGM3:	1298	0	0
MODE:	1	0
PPB:	497	0	0
PPB:	341	0	0
PPB:	254	0	0
PPB:	201	0	0
PPB:	165	0	0
PPB:	141	0	0
PPB:	124	0	0
PPB:	119	0	0
PPB:	115	0	0
PPB:	112	0	0
MODE:	1	1
UGM3:	107	0	0
UGM3:	104	0	0
UGM3:	103	0	0
UGM3:	101	0	0

still dropping on this side
19:50:41.810 -> PPB:	368	10	0
19:50:44.812 -> PPB:	367	10	0
19:50:47.875 -> PPB:	371	10	0
19:50:50.894 -> PPB:	372	10	0
19:50:53.908 -> MODE:	1	1
19:50:53.955 -> UGM3:	370	13	0
19:50:56.991 -> UGM3:	831	12	0
19:51:00.042 -> UGM3:	849	12	0
19:51:03.082 -> UGM3:	833	12	0

Thanks! Would be great to compare and know if something is different about the 118 version or whether it's the same for 117.

My datasheet has version V1.2-20200306, yours?
Is there a newer one?

Not that I've found anyway. The vendor didn't list a datasheet at all. So I just searched for it and the v1.2 seems all over the place.
Though if those version bytes are really dates, doth the v117 (2020-07-28) and v118 (2021-02-03) would be later revisions than this document.

On the back of the sensor I have SNH 12 02 40 16 36
Serial number, no clue what it means, although reading it backwards it looks partially like a date...

So judging by the footer of the datasheet, the manufacturer's site is aosong.com.
On the english product page there seems to be no datasheet and the chinese page has the v1.2 manual with the same date.
Serials wise I've got
SNI<PHONE_NUMBER>
SNJ<PHONE_NUMBER>
SNJ<PHONE_NUMBER>

So serial wise you seem to have at least two different batches. The SNJ number differs 1400 so pretty close.

I am preparing a 0.1.4 version to add a function uitn32_t getSensorDate()
if you print the number as HEX you get a date.
uint32_t dd = sensor.getSensorDate();
Serial.println(dd, HEX);

(implementation might change, just a helper function for debugging)

works
AGS02MA_LIB_VERSION: 0.1.4

BEGIN:	1
VERSION:	117
DATE:	28072020   <<<<<<<<<
VERS:	117
MODE:	1	0
PPB:	311	11	0

BEGIN:	1
VERSION:	117
DATE:	28072020
VERS:	117
MODE:	1	0
PPB:	680	11	0    <<<<  status = 1 0001 busy bit set - meaning not ready which we see in an wrong value
PPB:	301	10	0    <<<<  status = 1 0000 busy bit clear - meaning data ready)
PPB:	305	10	0
PPB:	304	10	0
PPB:	306	10	0
PPB:	314	10	0
PPB:	309	10	0
PPB:	304	10	0
PPB:	297	10	0
PPB:	294	10	0
MODE:	1	1
UGM3:	290	13	0  <<<<  status = 1 0011 busy bit set + mode bit set =>UGM3 mode
UGM3:	657	12	0  <<<<  status = 1 0010 busy bit clr + mode bit set =>UGM3 mode
UGM3:	654	12	0

so status field gives correct info.
Question is if the read function should return some error or so?
Need to think that one over...

It looks like the both

Sets the busy bit.
Returns the last known value of the previous mode.
So, possibly yes it might be good to error, since the readout isn't in the mode you'd expect, or what it's reporting!

So I've messed about with resistors, 3.3v and 5v, and an external 1A power supply. None of which made any difference, unsurprisingly. I really think either the protocol changed or something isn't working properly and the v118 exists for silicon shortage reasons xD

I have tested with an UNO which is 5V, you tested with ESP32 ?
The sensor works from 3V3 - 5V5 so it might be that the ESP just provides not enough "juice"?

I'm testing with the UNO currently.
The ESP-WROOM-32 I tried as well, but not using the Arduino IDE or your library. Wrote the I2C protocol with ESP-IDF referencing your library there and got the same results, no mode switching, 0x00 status byte.

It looks like the both

Sets the busy bit.
Returns the last known value of the previous mode.
So, possibly yes it might be good to error, since the readout isn't in the mode you'd expect, or what it's reporting!

The user can (should) check the ready bit after each mode switch with dataReady()
I can set error flag but the last read value of the previous mode might be useful so I do not clear it.
User needs to check the error or ready flag.

added
#define AGS02MA_ERROR_NOT_READY     -13

and code in .cpp to set this error flag.

Can you make a 2nd dump up to reg 22  to see what changed?
Most is calibration stuff I expect, but it might also have some counters or last measured value?...
known registers
#define AGS02MA_DATA                  0x00
#define AGS02MA_CALIBRATION           0x01
#define AGS02MA_VERSION               0x11
#define AGS02MA_SLAVE_ADDRESS         0x21

I've at least grouped them by what the sheet mentions.
I'll have look at doing comparisons over time next.
Sensor data
REG[0]	OK:1	Err:0	CRC:0	Buff:	0	0	0	4E	F5

Zero calibration data
REG[1]	OK:1	Err:0	CRC:0	Buff:	0	7D	2	C0	5B

Undocumented
REG[2]	OK:1	Err:0	CRC:0	Buff:	0	64	1	3A	9F
REG[3]	OK:1	Err:0	CRC:0	Buff:	3	E8	0	8C	40
REG[4]	OK:1	Err:0	CRC:0	Buff:	27	10	0	42	CE
REG[5]	OK:1	Err:0	CRC:0	Buff:	EE	48	23	28	CD
REG[6]	OK:1	Err:0	CRC:0	Buff:	1A	0	0	12	D5
REG[7]	OK:1	Err:0	CRC:0	Buff:	1	B4	2C	85	8A
REG[8]	OK:1	Err:0	CRC:0	Buff:	52	76	0	14	5

Version number
REG[11]	OK:1	Err:0	CRC:0	Buff:	15	2	3	76	85

Undocumented
REG[20]	OK:1	Err:0	CRC:0	Buff:	0	1	B4	82	9B

I2C slave address
REG[21]	OK:1	Err:0	CRC:0	Buff:	1A	E5	1A	E5	71

last version 0.1.4 shows explicit the not ready flag (in an uint32_t way)
BEGIN:	1
VERSION:	117
DATE:	20200728           <<<<< ISO date format
VERS:	117
MODE:	1	0
PPB:	0	11	FFFFFFF3    <<<<<<<  error = not ready
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	0	11	FFFFFFF3
PPB:	485	10	0
MODE:	1	1
UGM3:	0	13	FFFFFFF3
UGM3:	1062	12	0

....

PPB:	192	10	0
MODE:	1	1
UGM3:	455	13	FFFFFFF3
UGM3:	441	12	0
UGM3:	437	12	0
UGM3:	439	12	0
UGM3:	439	12	0
UGM3:	439	12	0
UGM3:	439	12	0
UGM3:	441	12	0
UGM3:	439	12	0
UGM3:	441	12	0
MODE:	1	0
PPB:	192	11	FFFFFFF3
PPB:	193	10	0

A first (manual) analysis of the hex data gave a few hints
REG      HEX              UINT32_T       UINT16_T
                                           A      B
========================================================
 2      0064 013A         6.553.914       100     314
 3      03E8 008C        65.536.140      1000     140
 4      2710 0042       655.360.066     10000      66
 5      EE48 2328     3.997.705.000     61000    9000
 6      1A00 0012       436.207.634      6656      18
 7      01B4 2C85        28.585.093       436   11397
 8      5276 0014     1.383.464.980     21110      20
 20     0001 B482           111.746         1   46210

The hex codes do not resemble ASCII
REG 2 - 4 look like a look-up table  in uint16 view
314 might be PI   314 / 100 ~ PI
others unknown
no obvious link to your serial numbers.

The above could be automated

you might try to interpret the data as a 4 byte float?

Back to work :)

Observing a bunch of these over time I found that registers 2-6 are static for me. 7, 8 and 20 do change.
8 and 20 only update if you pause long enough to update the (reg 0) measurement value. Some intermediate calculations?
Reg 7 seems to be two uint16 parts, the A part seems roughly inversely proportional to the reg 0 measurement value when I spike the reading with some gas. B part is all over the place, can't tell what that is, maybe some clock or counter.

I also learned you can get 3 degrees of readiness, depending on how much you pause between I2C commands.

Enough time to update Reg 0 (for example 1500ms pause between dumps).
Enough to update Reg 7, but not Reg 0, 8, or 20 (for example 500ms between dumps).
Only Reg 7's B part updates, while the rest stays fixed (for example 50ms between dumps).

Presumably the I2C commands just hogs all the internal chip's processing cycles, preventing any measuring. And the in between nr 2 state seems to indicate some internal samples being stored in Reg 7.
What that tells me is: my v118's busy bit is also not correct.
Not just the UGM3 mode, but at least in the nr 2 situation where it's able to take some samples but not enough to update Reg 0, it should have the CPU time to flag itself as "busy", but doesn't do so.

Probably time to try customer support for maybe an updated datasheet and why it's neither using the busy bit correctly and seems to be lacking the UGM3 mode entirely.

I have made a  request to find out if the datasheet is the latest (through my preferred shop who are reseller)

Thanks I also contacted the reseller I used and aosong.com contact emails, which I assume is the manufacturer's real site.

B part is all over the place, can't tell what that is, maybe some clock or counter.

if it is a clock it probably is micros ?
If you do 4 carefully timed reads you might calculate its step size.
So reg 7 can be the raw read of particles, it is multiplied with the constant registers to get uG as I expect it uses a constant factor between the two measurements.
mmm
maybe those constant registers are the molarity thingy ?
You might check the numbers for mass of some molecules?  (quick look doesn't ring any bell)
Is it possible to write them?

@Beanow
Just reading back the thread I don't think the problem with the status bit is caused by the library. we did not find a way to solve it either yet. So I will keep the issue open at least until there is more info about the datasheet.

Yes, it's definitely the sensor not behaving as advertised. Thanks for helping debug this, let's wait for the retailer/manufacturer response, as for now this revision just looks partially functional.

@Beanow
FYI, got no info so far.

Me either @RobTillaart
No response at all from the manufacturer, not even a confirmation they got my email.
The retailer promised an email with "the information", whatever that'll be, last week. Haven't gotten that email yet.
Apparently though

2022.01.26-2022.02.10 is the Chinese Spring Festival holiday

Perhaps that has something to do with it.

Guess so.

So, in the meantime. The retailer sent me the Chinese v1.2 datasheet. Even though I told them I already had the v1.2 and was wondering if there was a newer one.
It's a good thing I gave them a dedicated email address, because who would otherwise open an attachment like this?

Now the technical staff explains:

(With PPB underlined.)
Did they just silently downgrade to an AGS01DB or what? They have AGS02MA printed on though. So whatever this is it looks like ASAIR's shenanigans more than the retailer. Though the retailer removed the product I left a bad review on and posted it again.
Old: https://www.aliexpress.com/item/1005003131194702.html
New: https://www.aliexpress.com/item/1005003918818892.html
Still claiming ug/m3 output mode.
:thinking: seems pretty trustworthy.

interesting,
I recall that sometimes when production line changes from A to B there are products that are labeled incorrectly.
This happens to sensors, seen ADS1015 labeled as ADS1115,  I've even seen it with soup (wrong wrapper around tin).
But that said, it becomes unfixable and I will spend some lines about it in the readme
I won't change the library to handle such exceptions unless it becomes "mainstream".
(would imply to read the version on start and add version dependent conditions and return values in the code. That is a recipe for less maintainable code and increased footprint.)
So for now it seems a "won't fix" issue. If you have an idea for a workable solution, let me know.

Huh, haven't encountered it before. I'd have to dig a bit deeper into the protocol and such if it really is a AGS01DB under the hood, yeah it wouldn't make sense for the library to address as being a AGS02MA flavor.

@Beanow
going to add this to the readme.md. Is it clear enough?
Version problem?
The library can request the version with getSensorVersion().
My devices all report version 117 and this version is used to develop / test this library.
There are devices reported with version 118 which only support the PPB and not the ugM3 mode.
It is unclear if this is an incident, bug or a structural change in the firmware.
So if you encounter problems with setting the mode, let me know.
Also if you have a working device, not being version 117 or 118 let me know too.

LGTM, at your discretion could link this issue for context. But I think this should do.

@Beanow
As it looks like we won't find a solution shall we close the issue?

Yeah, if I do get any more insights I'll create a new issue / reopen then :]


check attachment before Detach

it might have been detached by ctx.ptys.Close()
Signed-off-by: Lai Jiangshan<EMAIL_ADDRESS>

LGTM


1234

Same as https://github.com/github/platform-samples/pull/367, another merge training PR.
you're welcome, as always. all the best. 👍
ret394


Share more code between inline html and html7 blocks

Fixes https://github.com/raphlinus/pulldown-cmark/issues/336.

@raphlinus this could be a good candidate for 0.5.3

I am fairly confident this shouldn't introduce any new problems, so I believe we can go ahead. Thanks for having a look anyway! :+1:


[Add] Social Share Button

Details
The proposed addition is to implement a new social share button that has a unique structure and displays all the social media icons for sharing when hovered upon.
On hover, a hidden panel containing various social media icons will be revealed, allowing users to easily share the content on their preferred social platforms.
Screenshots
https://github.com/Spyware007/Animating-Buttons/assets/101655716/aba96767-52ba-431e-ad67-2634244ccf75
Type of Contribution

[ ] Update to an existing Animated Button
[X] Adding a new Button
[ ] Resolving a bug
[X] Proposal to the Repository
[ ] Changes related to documentation or README.md
[ ] Other Changes

Checklist

[X] I have checked the existing issues
[X] I have read the Contributing Guidelines
[X] I am willing to work on this issue
[X] I am a GSSoC'23 contributor

@Nishusk has this issue been assigned?


Keep spack.lock file at the end of the job

This file can be useful to reproduce the spack build used for the benchmark run.

Did you mean spack.lock?

Yes, sigh.  Thanks.


Use kustomize add operation for setting insecure registry

https://github.com/openstack-k8s-operators/ci-framework/pull/785 adds the support for setting insecure registry on EDPM nodes.
It works for EDPM. In BM job, we used replace operation which does not work as the key edpm_container_registry_insecure_registries does not exists in CR.
We need to use add as used in EDPM in order to make it working.
Note: It also bumps the timeout for tcib jobs as new containers got added and we need more then an hour to build all containers.
As a pull request owner and reviewers, I checked that:

[x] Appropriate testing is done and actually running

/approve

@pablintino care to check that out once you're back? Thanks :)

/lgtm


chore: add DOCS.md metadata for woodpecker plugins

Hi, thanks for this plugin.
I would like to add this plugin to woodpecker plugin gallery. I already opened a PR towards woodpecker repo.
Maintainers asked me to add required metadata. So here we are.

@appleboy :wave: :)


Improve error message when wallet runs against syncing index

If you run a wallet command while the index is syncing, you'll get "error: wallet failed to synchronize with ord server", we should instead either hang, with a message like "waiting for ord server to sync", or give an error message that includes the server height and the bitcoind height.

If you run a wallet command while the index is syncing, you'll get "error: wallet failed to synchronize with ord server", we should instead either hang, with a message like "waiting for ord server to sync", or give an error message that includes the server height and the bitcoind height.

Looks silly but usefull, it took me sometime to understand that msg the first time I got it.

I have been running into this issue a lot when trying to create a lot of inscriptions in a structured way using a bash script and ord client.
Do you know if there a work-around to this problem? Or what is the best way to create inscription hierarchies without running into this?


List of url() instead of using patterns()

Django 1.10 removes the old patterns() way to define URLs. This complies with the deprecation notice recommendation:

adminplus/sites.py:51: RemovedInDjango110Warning: django.conf.urls.patterns() is deprecated and will be removed in Django 1.10. Update your urlpatterns to be a list of django.conf.urls.url() instances instead.

Does this work with 1.8/1.9?

@jsocol it does, I tested it under 1.9 at least. The deprecation notice didn't show up until now I think, but this was deprecated in Django 1.8: https://docs.djangoproject.com/en/1.9/ref/urls/#patterns

Great, thanks!


wrong P4 and P6 disk size

P4 and P6 are respectively large 32GB and 64GB.
See here: https://docs.microsoft.com/en-us/azure/storage/storage-premium-storage#scalability-and-performance-targets

This seems like a small (but important) contribution, so no Contribution License Agreement is required at this point. We will now review your pull request.
Thanks,
Microsoft Pull Request Bot

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@rossanov : Thanks for your contribution to the Azure documentation! The author, @azdocprs, has been notified to review your proposed change.

@yuemlu, Please review these proposed changes. Thanks

Revised.

Hi @yuemlu, To resolve this pull request, please add an appropriate comment so that automation can take appropriate action.  Thanks

Addressed, please close.


The library errors out with Deno 1.0.1 +

Describe the bug
When trying to run a bot with updated Deno, it won't start.
To Reproduce
Try to upgrade Deno to the latest version (ATM, 1.0.2) and start the bot.
Andreys-MBP$ deno run --allow-read --allow-net --allow-env --unstable mod.ts
error: Uncaught Error: Uncaught TypeError: Invalid URL.
    at WorkerImpl.#poll ($deno$/web/workers.ts:210:17)

A similar thing happens if I run the bot in Docker with Deno 1.0.1.
With 1.0.0, everything works as expected.
Expected behavior
The bot is to start and connect to Discord

Don't know if this is helpful, but I'm getting the error from a different location
Compile file:///home/forest/Documents/git/discord-game-engine/mod.ts
error: Uncaught Error: REQUEST_CLIENT_ERROR
      throw new Error(Errors.REQUEST_CLIENT_ERROR);
            ^
    at handleStatusCode (https://raw.githubusercontent.com/Skillz4Killz/Discordeno/v2/module/requestManager.ts:183:13)
    at Object.callback (https://raw.githubusercontent.com/Skillz4Killz/Discordeno/v2/module/requestManager.ts:132:9)
    at async processQueue (https://raw.githubusercontent.com/Skillz4Killz/Discordeno/v2/module/requestManager.ts:55:7)
error: Failed at the start script
error: Failed at the dev script

I am actually using Deno 1.0.2 right now it seems to be working fine.

@EternallLight My guess is that somewhere you are adding a using a url that is invalid. Was there something specific that you are doing while bot is starting up that could trigger that? Perhaps something in one of your event handler functions.
@AngelOnFira This is an interesting issue, but i don't think its related to the Deno version. My first recommendation would be to upgrade to v3 as there were a ton of improvements to the library. If you still see issues, I would check the ClientOptions being passed in if everything is accurate. I will add some logs in a commit later today so that it is possible to get detailed logs.

I have just managed to get the bot online after upgrading to v4 and fixing some type errors afterwards. Thanks for the response!

If you are here for @AngelOnFira's comment this issue will be of your interest.


vscode-extensions.charliermarsh.ruff: 2023.22.0 -> 2023.24.0

update project urls https://astral.sh/blog/announcing-astral-the-company-behind-ruff

Description of changes

Things done

Built on platform(s)

[ ] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[x] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 238380 run on aarch64-darwin 1

1 package built:

vscode-extensions.charliermarsh.ruff


Add 'blank' damage enricher

Basically extending the damage enricher so you can leave it blank, such as just [[/damage]], which would then retrieve from the item's damage formulas rather than any written input.
Would also be helpful if the item itself is passed to the roll hook.

This is probably a direction for a future version when we also add a To Hit enricher to aid with creating item descriptions.

I'm going to say we need to support multiple damage types for this first so we don't get failed outputs, so blocked by https://github.com/foundryvtt/dnd5e/issues/3357


docs: explain underlying technologies of docker.

[x] Namespaces
[x] Cgroups
[x] UnionFS

wrote the docs for this but just kept it to basics so it doesn't turn into linux workshop


update: remove two SA which does not seem valid (#1254)

Signed-off-by: Wen Zhou<EMAIL_ADDRESS>(cherry picked from commit c0e07092ecf591fd46ec2b94f93226aade9166c4)

Description

https://issues.redhat.com/browse/RHOAIENG-13411
How Has This Been Tested?

Screenshot or short clip

Merge criteria

[ ] You have read the contributors guide.
[ ] Commit messages are meaningful - have a clear and concise summary and detailed explanation of what was changed and why.
[ ] Pull Request contains a description of the solution, a link to the JIRA issue, and to any dependent or related Pull Request.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

/lgtm

i should have this closed 2 days ago, it is not need for main branch any more.
code already on main


Add payment badges

fixed in https://github.com/inttter/md-badges/commit/2eab20a835bc7aa4f5cfb958e5e1a4ce68a00e3b


Add Brier score overview in Jupyter notebook.

This is simply a documentation update and can be previewed in the Brier Scoring section at https://github.com/FootyStats/footy/blob/feature/briers_score/background.ipynb
As previously discussed about Brier scores it was originally published as a score between 0 and 2.  More modern implementations take a score between 0 and 1, but don't take into account multiple categories.  As we will have at least three categories (home win, score draw and away win) we need to use the original implementation.  This will be especially true when scoring the predicted number of goals (which will the probability of zero through six goals being scored by either team).
This PR is part of the work towards #4.

LGTM


Roll engine 8d3841705030..5c8147450db5 (1 commits)

https://github.com/flutter/engine/compare/8d3841705030...5c8147450db5
git log 8d3841705030..5c8147450db5 --no-merges --oneline
5c8147450 Clear layer cache in RasterCache::Clear (flutter/engine#6915)
The AutoRoll server is located here: https://autoroll.skia.org/r/flutter-engine-flutter-autoroll
Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md
If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

@liyuqian : New roll has been created by https://autoroll.skia.org/r/flutter-engine-flutter-autoroll

Auto-roller completed checks. About to merge.


linux/mac beta customer testing failing for release

while running post submit checks for release branch flutter-2.12-candidate.4, linux/mac beta customer testing consistently fail, relevant build logs are: https://luci-milo.appspot.com/ui/p/flutter/builders/prod/Linux beta customer_testing/130/overview and https://luci-milo.appspot.com/ui/p/flutter/builders/prod/Mac beta customer_testing/126/overview

This is happening because git branch is being calculated as:
GIT_BRANCH: * (HEAD detached at FETCH_HEAD)

I have learnt a lot about how recipes work, and understood a decent amount of the variables. After a lot of trials I realized that cloning the repo might be the best way to debug.  I haven't quite finished debugging yet, and I have documented the steps I have tried out so far at : https://docs.google.com/document/d/1vF2QNKIQgDrhupO4VYcislU--vKp2qQenJ3jZxdZ0ws/edit?resourcekey=0-t6Xya2ETEX7NU8vjpDYF4w#

cl submitted at https://flutter-review.googlesource.com/c/recipes/+/28563

merged in dfdfca1cc7aff0b27be8e9274f391a82f99f68de

Closing as obsolete, these tests are not running anymore on release candidate branches.


When an expected standard library file is not found, the interpreter panics.

When an expected standard library file is not found, eg. lib/range_enumerator.gb the interpreter panics.
Although this is an abnormal setup case, I think there should be a cleaner/user friendlier way of handling the requires, which would raise the standard InternalError.
Test case:
rm lib/range_enumerator.gb
./goby -i

@st0012 shouldn't there be a return in https://github.com/goby-lang/goby/blob/29e1190ef4270529a43f16e34a1d275caeb080ad/vm/thread.go#L82?
It doesn't fix the error, but I think execution should halt if the file can't be read.

Closed by #645.


Fix/unit tests

Build work in Android Studio.
NodeManager and MatcherApi is deprecated.
Thanks for your help.


[Bug fixed]Add null pointer checks when configure ExampleCoding

Fixed the bug of "Exception when there is only InputTfExampleConfig but no OutputTfExampleConfig(see issue #10 )" via adding null pointer checks in ExampleCoding.java
Add related test case for different situation of ExampleCoding

 Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


JGProgressHUD is not a valid configured external dependency

Cant build project
n-msk-macchs6-1:Tuist-Pokedex ruachs6$ tuist generate
Resolved cache profile 'Development' from Tuist's defaults
`JGProgressHUD` is not a valid configured external dependency
Consider creating an issue using the following link: https://github.com/tuist/tuist/issues/new/choose

My bad, should run 'tuist fetch'


Bad sad error when specifying invalid key

Description:
The bad sad error happens when compiling code which specifies an invalid field name for the table key. ballerina-internal.log is attached.
Steps to reproduce:
Specify a key which does not actually exist in the table record.
type TableRec record {
    readonly int a;
    string b;
    float c;
};

type TestTable table<TableRec> key(id);

Affected Versions: Observed when using slp3.
OS, DB, other environment details and versions: macOS Mojave 10.14.6, running ballerina build from the command line.
Related Issues (optional): None
Suggested Labels (optional): slp3
Suggested Assignees (optional):
ballerina-internal.log

This is fixed in the latest preview version SLP5. Hence closing the issue.
Additional information
type TableRec record {
    readonly int a;
    string b;
    float c;
};

type TestTable table<TableRec> key(id);

public function main() {
    TestTable tab = table [
      { a: 1, b: "B", c: 12.23 }
    ];
}

If you construct a table using the TestTable table type as given above, you'll get the following compilation error.
field name 'id' used in key specifier is not found in table constraint type 'TableRec'


[FEATURE] Last.fm scrobble support

I would like the option to enable/setup scrobbling to Last.fm
There is a second extenstion I have set up to do this but would love to have everything in one place. Link to extension for reference:
https://microsoftedge.microsoft.com/addons/detail/web-scrobbler/obiekdelmkmlgnhddmmnpnfhngejbnnc?hl=en-US

Hey @magnusv0s! We previously had LastFM scrobbling in YTM+ (the predecessor to SynQ) but removed it due to low usage and the fact that the Web Scrobbler has near-universal coverage and that people with both Web Scrobbler and YTM+ were getting double scrobbles. That said, I'm open to community contributions for this feature. The code for YTM+'s scrobbling can be found here:

https://github.com/mkossoris/YTM-Plus/blob/master/src/util/lastFM.ts
https://github.com/mkossoris/YTM-Plus/blob/master/src/content-script/changeEmitters.ts#L95-L124


[c13cl] Adding support for www.13.cl and rudo.video

IMPORTANT: PRs without the template will be CLOSED
Description of your pull request and other information
Adds extractor for www.13.cl and rudo.video.
13.cl is a TV channel from Chile, and it uses rudo.video.
With this PR, yt-dlp now supports (georestricted to Chile):
https://www.13.cl/en-vivo
https://www.13.cl/en-vivo-2
https://rudo.video/live/c13
https://rudo.video/live/t13-13cl
https://rudo.video/live/bbtv
Template

Before submitting a pull request make sure you have:

[x] At least skimmed through contributing guidelines including yt-dlp coding conventions
[x] Searched the bugtracker for similar pull requests
[x] Checked the code with flake8 and ran relevant tests

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

[X] I am the original author of this code and I am willing to release it under Unlicense
[ ] I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

[ ] Fix or improvement to an extractor (Make sure to add/update tests)
[X] New extractor (Piracy websites will not be accepted)
[ ] Core bug fix/improvement
[ ] New feature (It is strongly recommended to open an issue first)

No problem :) Thanks for your advice @seproDev
I have just split the class into two clasess, RudoVideoIE and RudoVideoLiveIE. Also, I used those _og_search_* methods.

@seproDev I have used @pukkandan 's commit and modified just a little. The line to obtain the m3u8_url must first check for the streamURL variable, and only if it fails, search for the source tag.
This is because with https://rudo.video/live/bbtv (this is not geo-restrictred) the source tag has a generic m3u8 that doesn't work. That site also has the streamURL variable and that's the url that works. So using one regex to search for either streamURL or source tag failed. It was returning the generic m3u8 from the source instead of the streamURL variable.
IIRC podcasts have the m3u8 in the soruce tag.
Everything else should be just like pukkandan's suggestion

@seproDev I pushed your suggestions, adding support for the MP3 podcast, including a test


[chore] update eslint plugins

fixes warning on installation
same as this one
https://github.com/ProjectOpenSea/seaport/pull/288

can you update the package-lock as well?

done

mind re-applying it? Sorry :D had a big upgrade.


use multiple stages in Dockerfile

The current Dockerfile leads to a 1.5 GiB container and this patch cuts that down to 300 MiB. There are a couple of things I changed, some more controversial than others:

multi-stage build
The build is now done in two stages: the first builds singularity, the next copies it from the first. This way, you don't end up with go installed in the main container. It also means some build tools are no longer required.

golang container
The build now uses golang container instead of installing go from source. The advantage is that we don't have to care about the details of installing go and don't need its dependencies. We could go further in this direction by also not installing singularity from source but instead use the latest "slim" container from here https://github.com/singularityhub/singularity-docker I didn't do this so far but it might be worth a thought.

apt-get steps
One "trick" that is done quite often on Dockerfiles is to bundle apt-get update, apt-get install and rm -rf /var/lib/apt/lists/* into a single step. The point of that is that every RUN step in docker creates a new layer. The final image size is the sum of all layer sizes, so downloading the package info in one layer and deleting it in a later layer still uses the full space of it. Bundling the steps into one saves a bit of space over all.

removed packages
I wasn't clear on which packages where meant for the user and which where just dependencies. I removed a lot of non-essential packages where I thought they are not needed (hoping you'd correct me if I removed too much). For example, flex and bison where maybe thought for VAL, but VAL could also be build in a separate build stage or installed as a binary or singularity image and doesn't need those packages then. The remaining non-essential packages are bash-completion, vim and wget to make working inside the container more pleasant. I also left python3-venv which I assume was meant for lab? Though I'm not sure if this make sense because running experiments from inside the container is not ideal.

1st stage packages don't make it to the 2nd stage, right? unzip, curl, and git likely all should be included in the 2nd stage as well.
As you point out, many of the previous ones are just there as defaults from FD's recommended list.

Yes, the first stage is thrown away after the build (or could be at least, maybe docker caches it internally). I added the packages to the second stage as well. But isn't installing both wget and curl overkill? I thought curl was just there for the command installing go.

You're right about curl. Generally, if a package is required for installing a wide variety of things (wget, pip, git all fall into that category), then it's probably worth having installed as part of the docker. Curl doesn't make the cut ;)

Ok, I removed curl again. Either way, the docker container is a way of tracking/testing your dependencies: when you add new features to planutils and you make sure that they also run in the docker, then the list of installed packages in the main docker container act as a list of dependencies.

Aye, exactly. Any suggestion on what we'd do for esoteric dependencies? I.e., those that only a particular package would require?
Imagine the planutils --setup-server-environment: It will need to install node.js and a host of other things. Just have those setup scripts as bash files that need root access to go?

I am generally very careful with running scripts with root access, specifically when they make changes to my system (install software, etc.). Even ignoring the security issue, if some assumptions about my system are not correct, it can mess things up (I don't use bash for example, so writing things in my .bashrc does nothing, others might not use apt to install software, etc.). My point is that I wouldn't want a script to install node.js for me. Downloading a binary or a container is fine, building a container on my system is also fine, if the build is somewhat self-contained like the one by singularity. But installing software packages is a different issue for me.
I think a way around this would be to generate a script and let the user execute it themselves. If the script is simple enough, they can look through it and check that things match their system before they execute it. Another option (especially useful for servers) would be to have multiple dockerfiles for specific use cases: one dockerfile that is meant to run as a server, one that is meant for people windows who can run docker but no VMs (although, I would make vagrant the recommended default for those who can run it). They point of the server-docker would then not be to set up your server, i.e., make changes to your server setup, but the docker image itself would run as the server. I think the use cases for running a docker image as a server and running it locally are so different, it makes sense to separate them into two docker files.

Much like the other issue, why not both?
I hear you on the hesitation for running planutils in sudo mode constantly. I like the idea of "Oh, you want to do that? Well it's a little admin'y, so here's a script you can run if you really want to."
Then, those scripts are precisely what can(/should) be used for the tailored images. I can imagine a Docker/VM build for developing FD (e.g., giving it to students in an AI course to implement their own heuristic as a project). One for the webserver version. One for running a variety of planners. Etc.
As a slight aside, the webserver setup is also one that many are asking for in their local environments. Not to host their own service, but rather to wrap their planners for connecting to other programs (the vscode and online editor plugins both support this as having localhost being the solver server).
Do you think the initial Linux setup should be placed in a common script, rather than mirroring the details in both the Dockerfile and Vagrant flavours?

Yes, that sounds like a good idea. So to make sure, we are on the same page, is this what you meant:

have a couple of scripts that set up different parts (install singularity, install node.js, etc.)
have different containers for different use cases (and a readme that explains which container is meant for what)
inside the setup script for each container, call the appropriate scripts
on a native installation (not in a container), don't call the scripts directly, but tell the user to run them, so they can check and possibly adapt them.

In that case, the details to iron out would be what/how many containers do you want and how should we group the scripts? For the first question, I guess it would make sense to open a new issue for each container (the discussion here went a bit off topic for the original pull request). The answer to the question which tasks to group in scripts probably comes naturally from that.

Yes, and.
This is a relevant subset of the planutils initial idea (which is obviously something in flux / up for debate):
$ planutils --setup-server-environment
Setting up a webserver to call the installed planners...

$ planutils --setup-development-environment
Installing common dependencies for building planners...
Installing common planning libraries...

I would change the behaviour to be something like this...
$ planutils --setup-server-environment
Installation script created: setup_server.sh.
Please review the script and run it with administrator privileges.

$ sudo planutils --setup-development-environment --force
Installing common dependencies for building planners...
Installing common planning libraries...

And then in the docker files for a specific setup, we just run the --force version (maybe sudo isn't required there). There's an open question as to where those scripts live: string in a json config? files shipped with planutils?
The other question that I see being worth answering is the scope of scripts. I feel as though they should be mapped to specific environments (like the two examples above) rather than components of environments (like a script for just installing node.js). That make sense?

That usage seems a bit complicated to me. Say I'm going to set up a server with some hosted planners. With what you are suggesting I'd first install pip, then use pip to install planutils, then use planutils to generate a script, then use that script to make the actual installation. I think a more straight-forward setup would be one where I download the script and run it. In this view, the direction would be flipped, instead of planutils installing the server environment, the script for the server environment would install planutils, node.js, singularity, etc. on my server. After this setup is done, I could use the installed planutils on the server to download the planner images I want to host on the server. This would be similar to the different containers/VMs: you don't use planutils to get the container but instead get the container directly and it contains planutils so inside the container you can use it to get planners, validators, etc.
This would put the server setup script on the same level as the containers: all of them would be different ways to give you a system ready to run planners, while planutils would be the way to get the planners in a unified way. I think then it also makes sense to offer the different setups in parallel: if you are on windows/mac and want solutions to planning tasks use the VM (or client-docker); if you are on a linux use the client-side installation script to install things natively; if you want to run a server host the server-docker or use the server-side installation script. Independent of that choice, you can afterwards use planutils to download and run planners and other tools.
To answer your question about the scope of the scripts: I think it would be good if we can split things up in a way that avoids duplication. So it makes sense to have a server-side setup script and call this from the server-docker, and have a client-side setup that is called from the client-side docker. But if both of them install singularity, then it would make sense to have just a single script for installing singularity. But the scripts could be a collection of scripts that are based on each other, so it wouldn't have to be a black-and-white decision.

Very intriguing. Never thought of that inversion.
Scripts live in the planutils repo or elsewhere? A common practice for quick-and-easy installation is to host a script someplace that you can immediately download and run. Something like this would install docker (I may have the sudo in the wrong place):
curl -fsSL https://get.docker.com | sudo sh
So it would be convenient to have the scripts accessible for quick setup that way.
Either way, I like the idea!

I like the idea of having them easily accessible but I think this is independent on where they are hosted. For example, you can have URLs point to the "raw" files in a github repository. Since the setup scripts and containers are likely to grow and change, I'd put them in a repository. I have no strong opinion on whether to keep them in this repository or create a new one for them.

Ok, I'd probably advocate an environments/ folder here in the planutils directory. Each sub-directory could contain README.md, setup.sh, and Dockerfile doing the obvious. Any objections/additions/modifications? I can give an initial one a go as an example...
Btw, I'll likely create an issue to cover this and point to the conversation here -- just so happened that it spawned out from a PR discussion that's long since been merged!

Three addotions: first, I'd recommend one issue per environment instead of one issue for all of them (makes it easier to discuss them). Then the word "and" in "and Dockerfile" seems wrong. Aren't some of the environments VMs instead of Docker containers and some are scripts only. That brings me to my last point:: if every environment is its own directory, (how) are they sharing common code (like the script for installing singularity)?

Fair enough.
Well a docker image for each environment, no? Additionally, a Vagrant file makes sense. Ultimately, they would be minimal wrappers to the script, which installs what's required (and thus potentially requires root).
Hrmz...hadn't considered it. If sourceing is used, then the scripts could appear separately for re-use.

As a grounding, there are 3 environments already identified:

server: useful both for turnkey deployment of solver in the cloud as well as running planners locally to connect to via API's (like the PDDL editors or other software with planners-in-the-loop)
development: useful for building/developing planners and planner-related technology
evaluation: useful for setting up everything required for evaluating (lab, API tools to fetch benchmarks, etc)

Ahh, I see, you use a slightly different definition of "environment" then I did. I thought of "server (docker)" and "server (native)" as two separate environments. But I now see the advantage of treating them as one and just offering multiple ways to run the environment. Not all combinations make sense, though:

server: docker and native installation seem useful. I'm not sure about vagrant here. I think some servers are started as VMs but I don't know if the systems running the VMs typically support vagrant.
development: native installation and vagrant seem like the way to go here. I'd say that docker is not meant for this use case, but it might still be an option for people running windows in a way that prohibits VMs.
evaluation: this only makes sense locally without the overhead of docker or a VM. The only useful experiments you could do on a VM/docker would be ones that measure performance-independent data and are still fast enough to run on a single machine. If there is a virtualized environment like this, it should come with a big warning sign.

As for the sharing issue: this is a bit tricky for Docker because it only copies the files in the working directory to the build context and does not support symlinks for this. There are some options of how to handle this mentioned here: https://stackoverflow.com/questions/39376786/docker-and-symlinks
I think the options "Build script", "Build from URL", and "Increase the scope of the image build context" would work for us. Each one comes with its own set of disadvantages.

Aye, definitely not all make sense. Even if I'd personally prefer to play with all of them in my Docker-in-Windows, it doesn't mean we need to ship versions that don't make sense. If we have a native script that can install, then that script can just as easily be run from within a Docker container.
I'm not sure we need Docker sharing -- only one it really makes sense for (as you point out) is the server one.

Makes sense. lets start with just writing each environment independently and live with some duplicated code. If the duplication gets too bad, we can still think about how to fix it then.

#7: Server environment
#9: Developer environment
#10: Evaluation environment


Improvement do not use bloom to replace in_or_bloom when rf need merge

Proposed changes
do not use bloom to replace in_or_bloom when rf need merge

run buildall

run buildall

run buildall


"url" parameter is valid but image type is not allowed

Hello team,
Thanks for your awesome contribution to this stack. I am facing one issue with the latest version of this stack. Here is an issue:
https://uat-webv3.whimstay.com/_next/image?url=%2Fimage%2Ficons%2Fguest.svg&w=48&q=75
Is anything needed to take care of in the latest version? I am not able to load some images. It's throwing an error of 400 in the network tab with the response '"URL" parameter is valid but image type is not allowed. '
Can you please help with this?
Thanks

Hi,
since the latest version uses the Image Optimizer of Next.js 12.1.0 SVGs are no longer automatically optimized.
To keep SVG optimized, you have to add the following configuration to the next.config.js:
module.exports = {
  images: {
    dangerouslyAllowSVG: true,
    contentSecurityPolicy: "default-src 'self'; script-src 'none'; sandbox;",
  },
}

However I see that we forgot to pass the new configuration properties to the underlying Image Optimization module.
So setting the values currently has no effect, will fix this in the next release.

Sorry, the important part of my previous message was a bit lost:

So setting the values currently has no effect, will fix this in the next release.

Got it. Waiting for your fix and release. As we have a blocker to not going to production release till your fix is released. Hope this will be released ASAP.

This has been fixed in v0.12.1, thanks for reporting!

Thanks for the quick release. It really helped me to check my issue. But sadly, the Issue still remains the same. not resolved.
Not able to load any SVG files on my website.
Please check: https://uat-webv3.whimstay.com/detail/sextant-or-habitat-brickell-or-10-mins-to-beach-or-gym-pool-or-penthouse-or-3-bed-2-bath/99620?name=Miami%2CFL&adultsCount=1&childrenCount=0&suitablePet=false&whimstayPlus=true&timestamp=1649701270473

Deployed with latest version.

next.config.js:
const withPlugins = require("next-compose-plugins")
const path = require('path');
const withBundleAnalyzer = require("@next/bundle-analyzer")({
  enabled: process.env.ANALYZE === "true",
})

const defaultConfig = {
  target: "serverless",
  swcMinify: false,
  webpack: (config, { buildId, dev, isServer, defaultLoaders, webpack }) => {
    if (!isServer) {
      config.resolve.fallback.fs = false;
    }
    // config.sassOptions = {
    //   includePaths: [path.join(__dirname, 'src/styles')],
    // };
    return config
  },
  eslint: {
    ignoreDuringBuilds: true,
  },
  experimental: {
    optimizeFonts: true,
    modern: true,
  },
  sassOptions: {
    includePaths: [path.join(__dirname, 'src/styles')],
  },
  redirects: require("./next-redirect"),
  images: {
    domains: ["******-bucket.s3.us-east-2.amazonaws.com", "****-bucket.s3.us-east-2.amazonaws.com", "********.cloudfront.net", '*******.cloudfront.net', 's3-us-east-2.amazonaws.com'],
    // formats: ['image/avif', 'image/webp'],
    //minimumCacheTTL: 60,
    //disableStaticImages: true,
    dangerouslyAllowSVG: true,
    contentSecurityPolicy: "default-src 'self'; script-src 'none'; sandbox;",
    //layoutRaw: true,
  },
}

module.exports = withPlugins([withBundleAnalyzer], defaultConfig)

Is anything missing from myend?

Did you also updated the tf-next builder in the package.json to the latest version?
Can you check the content of the .next-tf/config.json that is generated after running tf-next build?
The file is used as input for the Terrafom module and the values of dangerouslyAllowSVG & contentSecurityPolicy are taken from there.
So if the values are not present in this file this could indicate that you are using the wrong version of tf-next or that something is still broken on our end.


Release Google.Cloud.SecurityCenter.V1 version 3.21.0

Changes in this release:
New features

Add IServiceCollection extension methods for client registration where an IServiceProvider is required. (commit 022fab2)

Created release for Google.Cloud.SecurityCenter.V1-3.21.0


/ode/testDrift02.py AttributeError: RbParticleBeam6D instance has no attribute 'getPhaseSpace6D'

C:\Users\Win\src\radiasoft\radtrack>python ./test/ode/testDrift02.py
Traceback (most recent call last):
  File "./test/ode/testDrift02.py", line 57, in <module>
    mySpace = myBunch.getPhaseSpace6D()
AttributeError: RbParticleBeam6D instance has no attribute 'getPhaseSpace6D'

This non-test has been moved to the examples area.


Support multiple correct words

Currently, only one word is correct. However, some items have multiple correct names. We should accept all possible names.
This requires changes to the model and API.

Show all possible words at some point.

Fixed by Merge "Add alternative word models"


fix: ensure the shell function is always called

Two changes:

Ensure the shell function is always called using the function_call parameter on the completion
Handle messages that are function calls but don't have the function_call finish reason.

@searchableguy you probably don't need these anymore:

https://github.com/searchableguy/whiz/blob/main/src/completion.ts#L17
https://github.com/searchableguy/whiz/blob/main/src/index.ts#L55

Thanks for the pull request!


Style changes apply across navbars now

[x ] No discrepancies across browsers. Eg. chrome vs safari
[ x] Pages pass W3C Validation for HTML (https://validator.w3.org)
[x ] CSS (https://jigsaw.w3.org/css-validator/)
[x ] Google sign-in is operational
[x ] Accessibility functions
[x ] Pages can scale without distorting page
[x ] No dead links
[x ] Navbar is consistent across the site
[x ] Pages are screen-reader accessible
[x ] Contrast meets standards for accessibility
[x ] Pages are keyboard accessible
[x ] Passes WAVE Evaluation Tool chrome extension

Looks good, just needs some padding between links and google button


Zappa Installation: UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 2347

Context

I tried pip install zappa, but get the following error
ERROR: Command errored out with exit status 1:
 command: 'X:\Anaconda3\envs\py36\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\quany\\AppData\\Local\\Temp\\pip-install-gxsvadov\\kappa\\setup.py'"'"'; __file__='"'"'C:\\Users\\quany\\AppData\\Local\\Temp\\pip-install-gxsvadov\\kappa\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\quany\AppData\Local\Temp\pip-install-gxsvadov\kappa\pip-egg-info'
     cwd: C:\Users\quany\AppData\Local\Temp\pip-install-gxsvadov\kappa\
Complete output (7 lines):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\quany\AppData\Local\Temp\pip-install-gxsvadov\kappa\setup.py", line 54, in <module>
    run_setup()
  File "C:\Users\quany\AppData\Local\Temp\pip-install-gxsvadov\kappa\setup.py", line 22, in run_setup
    long_description=open_file('README.rst').read(),
UnicodeDecodeError: 'gbk' codec can't decode byte 0x80 in position 2347: illegal multibyte sequence
----------------------------------------

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Your Environment

Zappa version used: lastest
Operating System and Python version: Python3.6 & Python3.7
The output of pip freeze:
certifi==2019.11.28
wincertstore==0.2

@qy2205 just to confirm, this is on Windows?

having the same issue, have you solved this yet?

facing the same problem

@qy2205 just to confirm, this is on Windows?

having the same issue, have you solved this yet?

having the same issue, have you solved this yet?

I haven't solve that yet.

I have looked up online. It says that its because of the chinese system, but I still don't know how to solve it. At last I got the app deployed by installing zappa on amazon ec2.

@sgpikachu Could you please explain how you deployed the app on amazon ec2?

@amitfriedman93 Just upload the web application to ec2, install python, install the virtual env, and deploy it on zappa


Various

Coverage increased (+0.1%) to 51.733% when pulling 540ed44729715abf169f72eaec950d2acb19d251 on various into 18bcc27c1d8374f8ea85814ed770f17d08f96ae1 on master.


Fix some files License header

todo:Pay attention to dual protocol files under config

It seem your target branch is master?

It seem your target branch is master?

We need to maintain two branches. master&dev

Oh, I forgot  this point, thank for clarification. 👍

Oh, I forgot this point, thank for clarification. 👍

Thx,PTAL

hi @CalvinKirs, I have a question is the following version correspondence correct?
master: 1.x
dev: 2.x

yup, you are right

hi @CalvinKirs, I have a question is the following version correspondence correct? master: 1.x dev: 2.x

yup, you are right.