cog: Fix SRC_URI

It does not use v${PV} anymore and use BP instead of cog-${PV} while
here
Signed-off-by: Khem Raj<EMAIL_ADDRESS>

@aperezdc is this a recent change in the Cog release procedure?

@aperezdc is this a recent change in the Cog release procedure?

Ah, sorry, I didn't intend to remove the prefix for Cog but force of habit made me forget about adding the “v” (none of the other components use it). I have pushed the missing tags a minute ago.
Actually, I would expect people to use tarballs from https://wpewebkit.org/releases/ — which is our canonical download site.
The only reason why I have been posting the same tarballs in GitHub is because otherwise some people will pick the ones that GitHub autogenerates from tags, which must not be used because the GitHub folks may change at any time how they are generated so their hashes might change (and also their hashes do not match the ones published in release announcements).

This is fixed by f98f5c85e77d912c65501f3077ba17ce4e47d1f3


A few changes on top of dev-docs PR

Add syntax highlighting to Bash code blocks, and rearrange the TOC in the index for better flow.

Codecov Report

Merging #152 into dev-docs will not change coverage.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           dev-docs     #152   +/-   ##
=========================================
  Coverage     94.22%   94.22%
=========================================
  Files            40       40
  Lines          2442     2442
  Branches        464      464
=========================================
  Hits           2301     2301
  Misses           84       84
  Partials         57       57

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2ddf028...e9c9a61. Read the comment docs.


[Bug]: dialog handler doesn't catch "open with application" popup

Bug description
Open https://apps.apple.com/app/id1523054563?mt=8 with puppeteer
const handlePageDialog = (dialog: Dialog): void => {
  void dialog.dismiss();
};

page.on('dialog', handlePageDialog);

Dismiss doesn't work in this case as it simply doesn't handle this as a dialog.

Puppeteer version
16.1.1
Node.js version
v16.14.2
npm version
8.5.0
What operating system are you seeing the problem on?
macOS
Relevant log output
No response

Only the following dialog types are supported (alert, confirm, prompt, beforeunload). The dialog in the screenshot is a browser dialog and cannot be currently intercepted.

@OrKoN is it supported by Chrome DevTools Protocol? I couldn't find it.

You can find supported dialog types here https://chromedevtools.github.io/devtools-protocol/tot/Page/#type-DialogType


Do not generate concrete instance for concrete proxied components

Related problem
Since #704, proxies will use the proxy instance to call concrete methods, instead of delegating these method calls to a concrete delegate instance. This leads to concrete instances being used only for abstract methods. However, this leads to the redundant creation of concrete instances if the concrete component is proxied, as the delegate is never used in this scenario.
Requested feature
If the requested component is:

An explicit component (associated ComponentContainer)
Permits proxying
Is concrete
No attempt should be made to create a concrete instance to back the implementation, as it will not be used either way.

Raw instance is required for early pre-processing when proxies are not yet initialized, removing this would severely break early service providers


Saved Search: Add dialogporten api version

In order to support future breaking changes in backend all new saved searches should store which API it was created for.

Hello, I tried to solve the issue.
This is what I did:
Modified the SavedSearchInput and SavedSearchData types to include an apiVersion field. This change allows storing information about which API version was used when creating a saved search, supporting future breaking changes in the backend.
You can review changes in this commit: https://github.com/lattaai12/digdir-dialogporten-frontend-1384/commit/66fc3c04a99a29fd670e4763c79df82867eb2792.

[!CAUTION]
Disclaimer: The commit was created by Latta AI and you should never copy paste this code before you check the correctness of generated code. Solution might not be complete, you should use this code as an inspiration only.

This issue was tried to solve for free by Latta AI - https://latta.ai/ourmission
If you no longer want Latta AI to attempt solving issues on your repository, you can block this account.


es6 import should not emit a 'require' for d.ts file with const enum

TypeScript Version:  1.8.0 / nightly (2.0.0-dev.201xxxxx)
Code
example.ts:
// A *self-contained* demonstration of the problem follows...
import * as foo from './foo'
if (foo.Commands.Yay) {
   console.log("yay")
}

foo.d.t.s
export const enum Commands {
   Yay = 1,
   Boo = 2
}

Expected behavior:
foo.d.ts is  purely a typescript declaration. It doesn't have a foo.js file
when example.js is emitted it should have
if (1 /* Commands.Yay */){
   console.log("yay")
}

Actual behavior:
foo = require("./foo") // Should not be emitted for const enum uses

if (1 /* Commands.Yay */){
   console.log("yay")
}

duplicate of #4690


aws_cloudwatch_metric_data_point table limited for 3 hours.

Describe the bug
Hi @cbruno10 @misraved
I didn't succeed to fetch data from Cloudwatch for more than the default period.
the query is:
select * from aws_cloudwatch_metric_data_point where id='m1' and expression = 'SELECT AVG(MetadataNoToken) FROM "AWS/EC2" GROUP BY InstanceId' and timestamp >= now() - interval '7 day'
I am not able to fetch data from the history.
I am able to see only 3 hours ago
To reproduce
select * from aws_cloudwatch_metric_data_point where id='m1' and expression = 'SELECT AVG(MetadataNoToken) FROM "AWS/EC2" GROUP BY InstanceId' and timestamp >= now() - interval '7 day'
Expected behavior
I am trying to see more than 3 hours, for 30 days for example.
Additional context
Add any other context about the problem here.

@bigdatasourav
Yea, your query works! for older than 3 hours, but it's related only to one instance-id...

Hi @roisec , based on what we could find, when using an expression, there is a limit of only getting data for the last 3 hours according to Metrics Insights limits.
The queries that @bigdatasourav provided use MetricStats instead, which don't have the same limitation.
One suggestion that we haven't had a chance to try yet is using CloudWatch's ability to aggregate statistics for instances that have detailed monitoring enabled. This will require enabling detailed monitoring, which does come at an extra cost unfortunately, but you can try with only one, or a few instances, at first.
@roisec Can you please try enabling detailed monitoring on a few instances and then follow the example AWS CLI command in https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/GetSingleMetricAllDimensions.html, replacing the arguments like --metric-name, --start-time and --end-time (and possibly others based on your requirements)?
If that works (or doesn't work but looks like it should), can you please share the output of the command here, like you did above?
Thanks!

@cbruno10  I didn't succeed to see the data through mentioned command. (I don't see any data)
this is the command:
aws cloudwatch get-metric-statistics --namespace AWS/EC2 --metric-name MetadataNoToken --period 3600  --statistics "Average" --start-time 2023-05-30T15:15:00 --end-time 2023-05-31T17:15:00+05:30 --region us-east-1
Do you think I missed something?

@roisec Can you please confirm that you've enabled detailed monitoring for instances that you expect to see data for? Also, if you try a different --metric-name and/or start from a shorter duration, e.g., 2 hours, then 3 hours, then 12 hours, etc., are you able to get any data?


Implementing buildbot logfiles configuration

Improving debug of containers logs
While trying to help @tmacro on a issue this morning I found the option of logfiles in buildbot documentation and thought that it could be a nice to have in S3 CI.
documentation
You can checkout the full documentation about it here. But here's what you need to know:
logfiles
Sometimes commands will log interesting data to a local file, rather than emitting everything to stdout or stderr. For example, Twisted’s trial command (which runs unit tests) only presents summary information to stdout, and puts the rest into a file named _trial_temp/test.log. It is often useful to watch these files as the command runs, rather than using /bin/cat to dump their contents afterwards.
Personal thoughts
Checkout how this change affect the UI! You can see that you have your stdio field and now also have S3 and you will be able to follow up the logs while the command is running! Which could be useful.
IMHO this should be implemented implicitly eve side. But while the development is not done yet, we could have this working now.

Hello tcarmet,
My role is to assist you with the merge of this
pull request. Please type @bert-e help to get
information on this process.
Status report is not available.

Conflict
A conflict has been raised during the creation of
integration branch w/8.0/improvement/debug-s3-logs with contents from improvement/debug-s3-logs
and development/8.0.
I have not created the integration branch.
Here are the steps to resolve this conflict:
 $ git fetch
 $ git checkout -B w/8.0/improvement/debug-s3-logs origin/development/8.0
 $ git merge origin/improvement/debug-s3-logs
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.0/improvement/debug-s3-logs

@bert-e wait

Not ready for merge, as the work is not 100% done, I'm just getting your thoughts on this and with your approval on the idea I'll take the work forward

Great idea! 👍

This would be awesome!

Waiting for approval
The following approvals are needed before I can proceed with the merge:

2 peers

Peer approvals must include at least 2 approvals from the following list:

@RachedBenMustapha

@rahulreddy

@dora-korpar

Integration data created
I have created the integration data for the additional destination branches.

this pull request will merge improvement/debug-s3-logs into
development/7.4
w/8.0/improvement/debug-s3-logs will be merged into development/8.0 (pull request #1376)
w/8.1/improvement/debug-s3-logs will be merged into development/8.1 (pull request #1396)

The following branches will NOT be impacted:

development/6.4

Follow integration pull requests if you would like to be notified of
build statuses by email.

Waiting for approval
The following approvals are needed before I can proceed with the merge:

2 peers

Peer approvals must include at least 2 approvals from the following list:

@RachedBenMustapha

@rahulreddy

@dora-korpar

@bert-e reset

Reset complete
I have successfully deleted this pull request's integration branches.

Conflict
A conflict has been raised during the creation of
integration branch w/8.0/improvement/debug-s3-logs with contents from improvement/debug-s3-logs
and development/8.0.
I have not created the integration branch.
Here are the steps to resolve this conflict:
 $ git fetch
 $ git checkout -B w/8.0/improvement/debug-s3-logs origin/development/8.0
 $ git merge origin/improvement/debug-s3-logs
 $ # <intense conflict resolution>
 $ git commit
 $ git push -u origin w/8.0/improvement/debug-s3-logs

Waiting for approval
The following approvals are needed before I can proceed with the merge:

2 peers

Peer approvals must include at least 2 approvals from the following list:

@RachedBenMustapha

@rahulreddy

@dora-korpar

Integration data created
I have created the integration data for the additional destination branches.

this pull request will merge improvement/debug-s3-logs into
development/7.4
w/8.0/improvement/debug-s3-logs will be merged into development/8.0 (pull request #1400)
w/8.1/improvement/debug-s3-logs will be merged into development/8.1 (pull request #1401)

The following branches will NOT be impacted:

development/6.4

Follow integration pull requests if you would like to be notified of
build statuses by email.

My pleasure @rahulreddy !
Could be nice if we do the same for Zenko e2e tests, let me know if you guys need this and I'll do my best helping you out!

In the queue
The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.
The changeset will be merged in:

:heavy_check_mark: development/7.4

:heavy_check_mark: development/8.0

:heavy_check_mark: development/8.1

The following branches will NOT be impacted:

development/6.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of Release Engineering will
contact you to help resolve the matter.
IMPORTANT
Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of Release Engineering now.

In the queue
The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.
The changeset will be merged in:

:heavy_check_mark: development/7.4

:heavy_check_mark: development/8.0

:heavy_check_mark: development/8.1

The following branches will NOT be impacted:

development/6.4

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of Release Engineering will
contact you to help resolve the matter.
IMPORTANT
Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of Release Engineering now.

I have successfully merged the changeset of this pull request
into targetted development branches:

:heavy_check_mark: development/7.4

:heavy_check_mark: development/8.0

:heavy_check_mark: development/8.1

The following branches have NOT changed:

development/6.4

Please check the status of the associated issue None.
Goodbye tcarmet.


Failed to run subprocess in the console..

I want to run a program in the console and let the console show the result. However, I get an error as shown in the figure below. I use PIPE before and it works, but there is no output message....

@coldfix

I use PIPE and it can print the output, but the printing is not real-time..... It prints the result after the child process finishes......

Hi, thanks for your report.
This is due to overriding sys.stdout with a stream object that appends to our log widget. I don't think there is an easy fix. Some options would be:

implement Stream.fileno() to give a file descriptor for os.devnull, i.e. ignore output by suprocesses, but at least don't except
implement Stream.fileno() to return the file descriptor of the original sys.stdout, i.e. output might go somewhere (terminal/log file), but won't appear in the console widget
create a pipe object, read on it in a separate thread, and emit signals whenever we get a new line. This would allow appending the output to the console widget, but is also the most involved method by far. Also it might not work, because in windows pipe/socket/file descriptors are sometimes different and not always accepted by all respective APIs

Another option could be to expose an option or context manager to disable or revert the sys.stdout overriding, e.g. (user code in console):
with restore_io():
    subprocess.Popen(...)

The easiest solution for you as a user right now, is to not use Popen(stdout=sys.stdout), but specify an actual file, e.g. stdout=open('CONOUT$').


stress test with valid jwt

One of the concerns regarding impact-graph reliability is issues may happen through validating the jwt token in requests with the authentication server.
We must have a test routine/practice to load many requests with valid jwt on impact-graph in a short time and see whether they are successful or not.
The requirements of the test:

The test JWT must be valid
The requester/agent must be able to request impact-graph limitless (be excluded from impact-graph, nginx or caddy rate limit)
The test must be scheduled to be run regularly (once per day) and report the result if any issue happens.

FYI @mohammadranjbarz @CarlosQ96

@geleeroyale Let us know if you need any help from dev team or our testing specialist @maryjaf

Yea I will need help with creating the Jwt and validating it against the backend.

Yea I will need help with creating the Jwt and validating it against the backend.

@CarlosQ96 can you help them with it? Maybe creating a code snippet to build a valid JWT and introducing a simple query that needs validating JWT and doesn't require executing complex query on backend (e.g. getting like status).
@geleeroyale would it be helpful for you?

@mohammadranjbarz @mhmdksh Would you please collaborate on this? Actually, the rate limiting tests are not complete without having a scenario including valid JWT (an include requests between impact-graph and auth server)

Yea I will need help with creating the Jwt and validating it against the backend.

@CarlosQ96 can you help them with it? Maybe creating a code snippet to build a valid JWT and introducing a simple query that needs validating JWT and doesn't require executing complex query on backend (e.g. getting like status). @geleeroyale would it be helpful for you?

@geleeroyale  @mhmdksh  If you need any help regarding this issue, please let me know

@mohammadranjbarz Yes we need your presence for the stress tests and opinion about what specific scenarios we can do to push the performance limits as well as identify further bottlenecks. We can do that after pushing the changes of our backend stack to production

This was completed by @mohammadranjbarz - he shared some insights with us. If we want to incorporate stress test into our CI/CD pipeline, or have it on demand via GitHub actions - we can do that, but it should be a separate issue.
@aminlatifi we can close this one.

This was completed by @mohammadranjbarz - he shared some insights with us. If we want to incorporate stress test into our CI/CD pipeline, or have it on demand via GitHub actions - we can do that, but it should be a separate issue.
@aminlatifi we can close this one.

@geleeroyale Can devops team run stress testing with valid JWT without dev team support? Mohammad will be off the rest of the week.

@maryjaf steped forward to help with stress testing.
Maryam the DevOps team has conducted this stress test as well without JWT token AFAIK https://github.com/Giveth/devops/issues/179
This task aims to perform stress tests that cover various authentication situations as well, e.g. JWT token valid, invalid, without. The rationale of this variety is experimenting authentication service availability as well as impact-graph availability.

I've executed a request with jwt Token by postman performance test tool, and this is a result  of run with 40 virtual user in 2 mins

These look good to me, @maryjaf are the requests performed from different IP sources?? Please take into account that these are not done from our Whitelisted VPN. As it has a more relaxed rate limiting policy

Yeah, I run this test with outline server: on
and if run with vpn:off,  I got 429 (rate limit error )

Perfect tests @maryjaf
@jainkrati Taking into consideration the stress test values above. And knowing that we are targeting a specific number of users/minute. What do you think is the right rate limit value to be set in that case????

@maryjaf Was your jwt token valid? how did you create that? and use single one for all 40 virtual users?

Yeah it is a valid token and all request had a same authorization header,
I try to set a javascript code to generate different signature and after that different token but I haven't find a solution yet

What's the status of this issue? can we call it done?

Yeah it is a valid token and all request had a same authorization header, I try to set a javascript code to generate different signature and after that different token but I haven't find a solution yet, postman doesnt support web3 module

So, you may be able to generate multiple jwt tokens out of Postman and import them there. Do you think that will solve your issue?

Yeah it is a valid token and all request had a same authorization header, I try to set a javascript code to generate different signature and after that different token but I haven't find a solution yet, postman doesnt support web3 module

So, you may be able to generate multiple jwt tokens out of Postman and import them there. Do you think that will solve your issue?

Yeah, in previous test I generated token out of postman and run a request
I can do it with multiple jwt
but this token would be expired and if we wanna to have  a schedule run, the new file should be generated based on token lifetime

@maryjaf I found out Postman allows you to load a library from CDN, here. Web3 version 4 is downloadable from a cdn server (https://cdnjs.com/libraries/web3)

Yeah, in previous test I generated token out of postman and run a request
I can do it with multiple jwt

Report of running some request with 4 different jwt tokens :
Giveth-Collection-performance-report-11.pdf

@geleeroyale @mhmdksh to come back on estimate of date of completion fo rthis

@maryjaf can you close this issue if there are no errors you observed?


Update AB3 command

AB3 commands are updated to match the style of AWE

Codecov Report

Merging #165 (c15fb90) into master (36635b0) will not change coverage.
The diff coverage is 90.00%.

@@            Coverage Diff            @@
##             master     #165   +/-   ##
=========================================
  Coverage     60.14%   60.14%
  Complexity      503      503
=========================================
  Files           103      103
  Lines          1997     1997
  Branches        233      233
=========================================
  Hits           1201     1201
  Misses          739      739
  Partials         57       57

Impacted Files
Coverage Δ

...java/seedu/awe/logic/parser/AddressBookParser.java
73.07% <ø> (ø)

...a/seedu/awe/logic/commands/EditContactCommand.java
97.01% <75.00%> (ø)

...java/seedu/awe/logic/parser/EditCommandParser.java
92.30% <75.00%> (ø)

...va/seedu/awe/logic/commands/AddContactCommand.java
100.00% <100.00%> (ø)

...seedu/awe/logic/commands/DeleteContactCommand.java
100.00% <100.00%> (ø)

.../java/seedu/awe/logic/parser/AddCommandParser.java
100.00% <100.00%> (ø)

...va/seedu/awe/logic/parser/DeleteCommandParser.java
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36635b0...c15fb90. Read the comment docs.


HDDS-6312. Use KeyPrefixContainer table to accelerate the process of DELETE/UPDATE events

What changes were proposed in this pull request?
Recon stores the mapping of ContainerKeyPrefix in local RocksDB. When Recon is applying DELETE or UPDATE events from OM, it will run search the whole table for each to_be_deleted record.
In a big cluster, the record count in this table could be very large, and the search loop for each records is very slow. In our cluster there are 90m records, each loop cost over 70 seconds, if a delta OM events have 100 DELETE or UPDATE events, it will took about two hours to apply these updates.
This ticket is to accelerate the process with the help of a new local table.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6312
How was this patch tested?
unit test.

@adoroszlai @avijayanhwx @ferhui Could you help to review this PR?

@JacksonYao287 Could you please help review?

@avijayanhwx Could you help to check this issue?
This issue can be a problem if we want to retrieve updated information from OM.
In a small cluster, the connection from Recon to OM can be quite normal, with 10 minutes' interval. Please ignore the content in the following block, but each request is from Recon to OM to get delta updates.
2022-04-06 09:41:57,273 [Socket Reader #1 for port 9862] WARN org.apache.hadoop.ipc.Server: Connection Authentication from Recon:56662 for protocol org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol is failed for user ozone (auth:SIMPLE).
2022-04-06 09:51:57,357 [Socket Reader #1 for port 9862] WARN org.apache.hadoop.ipc.Server: Connection Authentication from Recon:60932 for protocol org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol is failed for user ozone (auth:SIMPLE).
2022-04-06 10:01:57,436 [Socket Reader #1 for port 9862] WARN org.apache.hadoop.ipc.Server: Connection Authentication from Recon:63884 for protocol org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol is failed for user ozone (auth:SIMPLE).
2022-04-06 10:11:57,544 [Socket Reader #1 for port 9862] WARN org.apache.hadoop.ipc.Server: Connection Authentication from Recon:24028 for protocol org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol is failed for user ozone (auth:SIMPLE).
2022-04-06 10:22:04,916 [Socket Reader #1 for port 9862] WARN org.apache.hadoop.ipc.Server: Connection Authentication from Recon:26968 for protocol org.apache.hadoop.ozone.om.protocol.OzoneManagerProtocol is failed for user ozone (auth:SIMPLE).

But in a big cluster, since the process is too slow, this connection can be far longer then 10min's interval.

Thanks @symious for the patch and sorry for the long delay in review.  Can you please resolve merge conflicts?

@adoroszlai Thanks for the review. Updated the patch, please have a check.

Thanks @symious for the patch. There's another patch causing minor conflicts. Would you merge latest master to the PR branch again?
I have resolved the conflict on my branch (I tried to push to your PR branch but got no permission), you can take this as a reference: https://github.com/smengcl/hadoop-ozone/blob/HDDS-6312/hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/tasks/ContainerKeyMapperTask.java#L249-L275

@smengcl Updated the PR, please have a look.

@smengcl Updated the patch, please have a look.

@smengcl Thank you for the review.

@smengcl Do you have any suggestions on the failed unit test?

@smengcl Do you have any suggestions on the failed unit test?

The failure looks related at a first glance: https://github.com/apache/ozone/runs/8008158105
Error:  org.apache.hadoop.ozone.recon.recovery.TestReconOmMetadataManagerImpl.testUpdateOmDB  Time elapsed: 0.249 s  <<< FAILURE!
java.lang.AssertionError
	at org.junit.Assert.fail(Assert.java:87)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertTrue(Assert.java:53)
	at org.apache.hadoop.ozone.recon.recovery.TestReconOmMetadataManagerImpl.testUpdateOmDB(TestReconOmMetadataManagerImpl.java:139)

But Line 139 in TestReconOmMetadataManagerImpl.java doesn't really match any meaningful code on your branch, hmm.
I ran the same test locally and it passed.
Retriggering the CI.

@symious , question -- what is the reason for adding KeyPrefixContainer but not reusing ContainerKeyPrefix?  I would like to see if we can get ride one of them.

In a big cluster, the record count in this table could be very large, and the search loop for each records is very slow. In our cluster there are 90m records, each loop cost over 70 seconds, if a delta OM events have 100 DELETE or UPDATE events, it will took about two hours to apply these updates."

@szetszwo It is to solve the slowness of processing.

I understand that this is a to fix the slowness.  Well done!
My question is why we cannot reuse ContainerKeyPrefix?  How KeyPrefixContainer can fix the slowness but  ContainerKeyPrefix cannot?

My question is why we cannot reuse ContainerKeyPrefix? How can KeyPrefixContainer fix the slowness but ContainerKeyPrefix cannot?

The slowness was coming from the operation of "getContainerForKeyPrefixes", since ContainerKeyPrefix is for mapping "container -> keyPrefix", so if we are trying to get the containers for a specific keyPreifx, we need to travel all the table of ContainerKeyPrefix, and it's time consuming when ContainerKeyPrefix is large.
With the help of "KeyPrefixContainer", the operation of "getContainerForKeyPrefixes" will be quite fast.
@szetszwo

@symious , sorry, I still cannot understand.  I probably have not asked the right question.  These two classes (shown below) look the same.  What we need to add KeyPrefixContainer but not reusing ContainerKeyPrefix ?
public class ContainerKeyPrefix {

  private long containerId;
  private String keyPrefix;
  private long keyVersion = -1;

public class KeyPrefixContainer {

  private String keyPrefix;
  private long keyVersion = -1;
  private long containerId = -1;

With the help of "KeyPrefixContainer", ...

How it helps?

@szetszwo Oh, I got your question.
I think I created a new class for the readability.
It is kind of confusing to have
  private Table<ContainerKeyPrefix, Integer> containerKeyTable;
  private Table<ContainerKeyPrefix, Integer> keyContainerTable;

...  for the readability.

If the code are the same, it is better to reuse the code.
//SCMMetadataStoreImpl
  private Table<BigInteger, X509Certificate> validCertsTable;
  private Table<BigInteger, X509Certificate> validSCMCertsTable;
  private Table<BigInteger, X509Certificate> revokedCertsTable;

For examples, we won't create a different BigInteger and a different X509Certificate for the tables above.

If the code are the same, it is better to reuse the code.

Reasonable to me, also.


changed-SelectProps-to-slotProps-in-docs

Fixes issue #41062
What does this PR do?
changes the deprecated TablePagination SelectProps to slotProps in docs

[x] I have followed (at least) the PR section of the contributing guide.

Hey @petsdom! Thanks for the contribution.
Sadly, another contributor had already asked about the issue and opened a PR, which has been merged: https://github.com/mui/material-ui/pull/41186.
I'll gladly review any other PRs you wish to work on. Remember to check if anyone is working on it beforehand. Have a nice day.


Feature pullapprove

#6

Pull Request Test Coverage Report for Build 28

6 of 6 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+0.3%) to 87.168%

Totals

Change from base Build 9:
0.3%

Covered Lines:
122

Relevant Lines:
128

💛  - Coveralls

Approved


Activate OS X builder on Travis-CI

And for some reason, the build failed: https://travis-ci.org/dask/dask/builds/218509271

Two blockers here:

we need to find a way to install graphviz on the OS X builder (shouldn't be too hard, hopefully)
OS X builds are still slow: the monthly backlog graph at https://www.traviscistatus.com looked encouraging, but that was really because the backlog was horrible at peak hours in the past, while it's simply large nowadays...

I think that the conda recipe for graphviz in defaults has improved.  It
might be worth trying.
If it slows down our ability to merge PRs I'm still comfortable waiting on
this.
On Wed, Apr 5, 2017 at 2:57 AM, Antoine Pitrou<EMAIL_ADDRESS>wrote:

Two blockers here:

we need to find a way to install graphviz on the OS X builder
(shouldn't be too hard, hopefully)
OS X builds are still slow: the monthly backlog graph at
https://www.traviscistatus.com looked encouraging, but that was really
because the backlog was horrible at peak hours in the past, while it's
simply large nowadays...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/dask/dask/pull/2164#issuecomment-291771717, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszJ9ROJfEcKNAX2tnmQW4gEpXxzI-ks5rszttgaJpZM4MzBF1
.

I've changed the OS X build to be marked "allow_failures". If I understand correctly, this should allow the whole build to be marked successful before the OS X build finishes.

This looks good to me.  Semi-relatedly, thoughts on removing some of the existing linux builds?  Perhaps the following:
    - PYTHON=3.6 NUMPY=1.11.2 PANDAS=0.19.2 COVERAGE='false' PARALLEL='true' TEST_IMPORTS='true' XTRATESTARGS=
    - PYTHON=2.7 NUMPY=1.11.0 PANDAS=0.19.2 COVERAGE='false' PARALLEL='true' TEST_IMPORTS='false' PYTHONOPTIMIZE=2 XTRATESTARGS=--ignore=dask/diagnostics

Though obviously that doesn't have to happen here.  I'm +1 on this.  Thanks for going through the trouble.

Yes, we have a lot of builds. Perhaps we don't need to test all those combinations...


Rename ADDRESS to OFFICE

Currently, the command format for add and edit uses a/ADDRESS to add in addresses for contacts.
This forces ASSETS to use A/ASSET (i.e capital A) instead, leading to inconsistency in prefixes and making it it take longer for users to enter commands.
Let's change a/ADDRESS to o/OFFICE, and A/ASSET to a/ASSET.

Dropped as using s/SERIAL_NUMBER instead

Added back, as using assetname instead of serial number


problem with sum type and focus

libs
I am using newest monocle
"dev.optics" %% "monocle-core" % "3.1.0",
"dev.optics" %% "monocle-macro" % "3.1.0",

with single import
import monocle.syntax.all._

and scala 2.13.10
problem
I would like to change email when PaymentMethod is PayPal (equivalent of copy)
val modified = user.copy(paymentMethod = user.paymentMethod match {
  case PayPal(_) => PayPal(newEmail)
  case dc@DebitCard(_,_,_) => dc
})

with quicklens I do simply
user.modify(_.paymentMethod.when[PayPal].email).setTo(newEmail)

and everything works, I would like to have something similar with monocle
code
case class User(name: String, paymentMethod: PaymentMethod)

sealed trait PaymentMethod
object PaymentMethod {
  case class PayPal(email: String) extends PaymentMethod

  case class DebitCard(
                        cardNumber: String,
                        expirationDate: YearMonth,
                        securityCode: Int
                      ) extends PaymentMethod
}

first try:
user.focus(_.paymentMethod.as[PayPal].email).replace(newEmail)

compilation error
value as is not a member of shared.user.PaymentMethod

another try:
user.focus(_.paymentMethod.focus().as[PayPal].email).replace(newEmail)

I get compilation error
value email is not a member of monocle.AppliedPrism[shared.user.PaymentMethod,shared.user.PaymentMethod.PayPal]

when I do something like this
val result: PaymentMethod = user
  .focus(_.paymentMethod.focus().as[PayPal].modify(_.focus(_.email).replace(newEmail)))
  .get

I get PaymentMethod as a return instead of User
however when I create everything by myself I get what I need
val newEmail =<EMAIL_ADDRESS>val setPaymentMethod = GenLens[User](_.paymentMethod)
val setPayPalEmail = Prism.partial[PaymentMethod, String]{case PayPal(x) => x}(PayPal)
val updatePayPalEmail = setPaymentMethod andThen setPayPalEmail
val result: User = updatePayPalEmail.replace(newEmail)(user)

how to do the same with focus?

@machisuji you would need to be using scala 3 to generate this optic.
Focus in scala 2 is much more limited. You can only using to generate lenses for fields inside a case class.


Your h2 should be red

Challenge Change the Color of Text has an issue.
User Agent is: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.79 Safari/537.36 Edge/14.14393.
Please describe how to reproduce this issue, and include links to screenshots if possible.
copy n paste the code, substitute blue for red, error - your h2 should be red.
reset code, my code goes away, but your h2 should be red remains.
I have signed back in, logged off, reset, but I cannot progress I am locked in.

My Code:

CatPhotoApp

Kitty ipsum dolor sit amet, shed everywhere shed everywhere stretching attack your ankles chase the red dot, hairball run catnip eat the grass sniff.

CatPhotoApp

@Pantera09 Thank you for reporting this issue. You don't need a new element. Just change the exiting h2 element.  Please reread the instructions.

Change your h2 element's style so that its text color is red.

This is a standard message notifying you that this issue seems to be a request for help. Instead of asking for help here, please click the "help" button on the challenge on Free Code Camp, which will take you to the help chatroom for that specific challenge. You can also view our full list of official chatrooms.
If you think I'm wrong in closing this issue, please reopen it and add further clarification. Thank you, and happy coding!


Wrong paper references for Ranger optimizer variants

The README lists Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM by Tong, Liang, and Bi (2019) as the source paper accompanying the Ranger, RangerQH, and RangerVA codes. However, this paper seems to describe only the addition of softplus to Adam (SAdam) and AMSGrad (SAMSGrad), implemented here: https://github.com/neilliang90/Sadam; it makes no mention of the LookAhead or RAdam techniques. Therefore it makes sense to credit RangerVA to this paper, but Ranger and RangerQH should not use this reference.
The original Ranger optimizer is a combination of the LookAhead, Rectified Adam, and Gradient Centralization papers, and is described in a blog post.
RangerQH uses quasi-hyperbolic momentum introduced by Ma and Yarats (2018) on top of the regular Ranger optimizer, so I believe this should be the reference.
I would propose the following references:

Ranger - New Deep Learning Optimizer, Ranger: Synergistic combination of RAdam + LookAhead for the best of both (2019) [https://medium.com/@lessw/new-deep-learning-optimizer-ranger-synergistic-combination-of-radam-lookahead-for-the-best-of-2dc83f79a48d]
RangerQH - Quasi-hyperbolic momentum and Adam for deep learning (2018) [https://arxiv.org/abs/1810.06801]
RangerVA - Calibrating the Adaptive Learning Rate to Improve Convergence of ADAM (2019) [https://arxiv.org/abs/1908.00700v2]

Would you like to submit PR with proposed changes?


feat: new version that supports printing the full version of an acronym

I added a function (and its plural version) to display the expanded / full form of an acronym, without messing with state.
Sometimes the expanded use of an acronym is useful to avoid constant repetition of the short form, in which case an explanation (expanded form + short form) is not needed.
#reset-acronym(...) almost allows this behaviour, but a) appends the short form as explanation and b) is less convenient as it requires two statements.
As such I would appreciate if the PR was accepted, as I currently use a modified version locally.
I am not sure about best practices, but I already placed it in a new version folder, so the changes to the file are a bit hard to trace. The concrete changes are:
#let acrfull(acr) = {
  //Intentionally display an acronym in its full form. Does not expand it and does not update state.
  [#display-def(plural: false, acr)]
}

#let acrfullpl(acr) = {
  //Intentionally display an acronym in its full form in plural. Does not expand it and does not update state.
  [#display-def(plural: true, acr)]
}

Thanks for the PR.
Sorry for the wait, I rarely come on GitHub.
I will take a look as soon as possible.


Updated HTTPServerResponse docs

Updated the docs by adding code examples to the HTTPServerResponse class

Codecov Report

Merging #259 into master will decrease coverage by 0.02%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #259      +/-   ##
==========================================
- Coverage   74.81%   74.79%   -0.03%
==========================================
  Files          31       31
  Lines        4181     4181
==========================================
- Hits         3128     3127       -1
- Misses       1053     1054       +1

Flag
Coverage Δ

#CHTTPParser
57.52% <ø> (ø)
:arrow_up:

#KituraNet
74.79% <ø> (-0.03%)
:arrow_down:

Impacted Files
Coverage Δ

Sources/KituraNet/HTTP/HTTPServerResponse.swift
78.84% <ø> (ø)
:arrow_up:

...ces/KituraNet/Server/ServerLifecycleListener.swift
79.16% <0%> (-4.17%)
:arrow_down:

Sources/KituraNet/HTTP/HTTPServer.swift
79.88% <0%> (-1.12%)
:arrow_down:

Sources/KituraNet/IncomingSocketHandler.swift
79.45% <0%> (+0.91%)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 49c39ad...21497b7. Read the comment docs.


Regex for multiline svg tags

I had an svg that started like this and props weren't being passed into the component.
<svg
  xmlns="http://www.w3.org/2000/svg"
  viewBox="0 0 40 40"
... etc
>

The current regex /(?<=<svg.*?)(>)/i uses .* which doesn't match newlines. I changed it to [^>]* to match everything except the closing bracket.
I also had trouble understanding the lookbehind, so I changed it to use a capture group and $1, which I think is easier to understand.
Here are links to text the regex against inputs
Current regex
https://regexr.com/7jor9
This pull request
https://regexr.com/7jorc

Thank you very much for the fix, and sorry for the delay.


AutoPlugin compatible

AutoPlugin
More test coverage(However, it covers only the basic route)
Version bump (0.6.0-SNAPSHOT)

The remaining work is to enrich the document

It seems better to fix the config related a little more, so I will close this PR one time.


Test local development of crud-web-apps/jupyter

In order to push features upstream, need to be able to locally run the Jupyter Web App in crud-web-apps to see if added features are working properly.

Steps that worked for me for a (stable enough) local development:

Clone master in StatCan/kubeflow
Follow the README to run frontend and backend. Had to make a few changes such as:

In the Frontend steps, when building the app frontend, the cd ../../../jupyter/frontend led to the wrong directory, right command is cd ../../../../../jupyter/frontend

In the Backend steps, make -C backend install-deps command didn't work for me, just run make install-deps

In the Backend steps, make -C backend run-dev would give me an error:
TypeError: '<' not supported between instances of 'int' and 'time.struct_time'
To fix this, I had to make this change https://github.com/kubernetes-client/python-base/issues/84#issuecomment-475216964 to my kube_config.py file (found in crud-web-apps/jupyter/backend/web-apps-dev/lib/python3.7/site-packages/kubernetes/config/) - I don't know why the merged PR's fix wasn't already present in this file.

After running all this, you can navigate to http://localhost:5000/ for a stable Jupyter Management UI

When I followed the README in jupyter/frontend and ran ng serve then navigated to http://localhost:4200/, I got many errors and it wouldn't let me connect to any namespace (the default was kubeflow-user). Just to get this to work, I tried to proxy to the backend like we did in our StatCan/jupyter-apis repo by adding the proxy.config.js file in crud-web-apps/jupyter/frontend and ran ng serve --proxy-config proxy.config.js. This then gave me access to the namespaces and notebook servers at http://localhost:4200/.

Still testing out this local development to see what will happen if I make changes and add in features, but that's all the progress I've made so far.

When trying to launch a new notebook server, get the following error:
[422] Notebook.kubeflow.org "test-jwa" is invalid: spec.template.spec.containers.resources.requests.cpu: Invalid value: "number": spec.template.spec.containers.resources.requests.cpu in body must be of type string: "number" http://localhost:5000/api/namespaces/saffa-alvi/notebooks


Predicting component contributions

This is something that crossed my mind when playing around with GPflow. Currently there is only support for predicting from the complete model using predict_f or predict_y. However, often in practical experiments the GP covariance function is a combination of many kernels explaining various aspects of the latent structure (say, a trend and periodic variation). It would be handy to have support for either:

the current prediction function to accept an optional component parameter, or
have a separate predict_components function

This is a rather small addition, but might make the software more useful for practical analysis.

I've been looking in to this a bit.
The AutoFlow is confusing me a bit. There doesn't seem to be a way to pass general objects through AutoFlow? In this case one would like to pass an instance of Kern.
Perhaps on can make a method to specify a component to leave active, then define a method predict_active_component, but it seems messy with coupled methods like that.

Hi @vals
I'm sorry you find the AutoFlow class confusing. You're right that it's not possible to pass in things that are not numpy arrays. In fact, that's the main reason that I've not fixed this ticket!
The problem is this. When we call an autoflow function, the first thing it does is execute the function being wrapped to get the tf.Tensor representing the return object. Then, we store that as storage['tf_result']. On subsequent calls, we don't re-run the function, which saves a lot of computation.
If we have a kernel as an AutoFlow argument, it's not clear what to do. Suppose you implemented predict_components, and then called  it:
k = GPflow.kernels.Matern32(1) + GPflow.kernels.RBF(1)
m = GPflow.gpr.GPR(X, Y, k)
mu, var = m.predict_component(Xtest, my_kern.matern32)
mu, var = m.predict_component(Xtest, my_kern.rbf)

Should we re-run the tf-function on the second call? How should we know that the kernel has changed? I think this is a little bit challenging.
One solution would be to use a function generator. The syntax would look like
m = GPflow.gpr.GPR(X, Y, k)
my_function = m.predict_component(my_kern.matern32)
mu, var = my_function(Xtest)

my_function is an instance of AutoFlow, which wraps (another) dynamically generated function. I don't find this very pleasing.
Comments and further suggestions welcome.

I'll think a bit about it
While I have you on the line, do you happen to know how GPy behaves with m.predict(X_new, kern=my_kern)? Is it just using my_kern also for m.X, or is it using m.kern in the prediction as well? I got a bit lost trying to figure it out from the code..
In pseudocode:
my_kern(X_new, m.X) m.kern(m.X, m.X)^-1 m.Y

vs
 my_kern(X_new, m.X) my_kern(m.X, m.X)^-1 m.Y

The former.
The reasoning is that the equation (without two kernels) is K_*f K_ff^-1 y, or rather K_*f [K_ff + \sigma^2]^-1 y. This is a contraction of K_*f K_ff^{-1}  mu where mu is the posterior mean of the function at the datapoints (f), which is given by (exercise for the reader).
This all comes from the equation int p(f* | f) p(f | y) df. When predicting with a new kernel, we assume that we're doing some 'inter domain' prediction, so we want to compute int p(g* | f) p(f | y) df. If you have a sum of kernels, you might assume that you have
f(x) = g(x) + h(x)
and you observe f(x) with noise, and now you want to predict g(x). The equation I just described is the correct way to do this.

Hello,
I started #491 for a similar reason as this ticket here exists. Would it be possible to give every kernel element an optional name argument so that you can then refer to it later, e.g. for the purpose of predicting component contributions? It would be great if you then could create models for prediciting the component contributions by handing in the component kernel via something like kern=m.kern['name'] or similar that would copy the parameters. The example given by @mathDR requires  to copy a lot of parameters from one model to another.
Thanks a lot!
Christian

This might, in fact, be a nice application of #398, where using a kernel as a function parameter should be perfectly possible and I think everything should "just work". I hope I will find the time during the next 2 weeks or so to try and implement a small example.

I have tried adding a predict_f_partial-function to GPR today and things are actually quite straightforward using the more flexible AutoFlow I think. Basically, after adding a parameter to build_predict, I created a function
@AutoFlow(Xnew=(float_type, [None, None]))
def predict_f_partial(self, Xnew, full_cov=False, kern=None):
    return self.build_predict(Xnew, full_cov, kern)

and can then call
mu_cos, _ = model.predict_f_partial(Xt, kern=model.kern.cosine)
mu_lin, _ = model.predict_f_partial(Xt, kern=model.kern.linear)

to predict the different kernel components separately, at least for the additive kernel example @jameshensman described above. Note however that I did not test this very thorougly.
More details and a few plots can be found here: https://gist.github.com/mrksr/6f020d6ce75ece9e0e1df16df21ef47a
Most of the code is just copied from GPR - the only thing changed in build_predict is the new parameter and the first few lines.

The code by @mrksr looks like a good interim solution, but after refactoring for things like the new autoflow decorator I'm getting a predict_f_partial() missing 1 required positional argument TypeError. Perhaps it has something to do with the 1.0 release changes.

The PR I did last year is probably not compatible to the new autoflow - but I have to admit I did not check because other things have been more important. If there is still interest in supporting a more flexible autoflow (and if it still makes sense) I can update the PR. It should be much more simple now that we do not have to support python 2.7.
@awav Any thoughts?

Hi @mrksr
Did you ever get round to updating this to work with the current gpflow build?
Thanks

Hi @henrymoss, I did not update the PR because I am not sure anymore it is the right direction to go. Tensorflow has progressed quite a bit (with things like eager execution), so the long-term perspective for GPflow is probably not to make autoflow more powerful but to instead get rid of autoflow at some point. The main reason for going into this direction is the interoperability with other tensorflow libraries. Please note that this is just my opinion and I do not "speak for" GPflow. A good place to discuss this might be #824. If there is interest to update the PR I am certainly willing to do it.
For my own work, currently I usually use the newish decorators (like gpflow.params_as_tensors) and do the rest in tensorflow.

Are there any working versions of this feature at the moment? Its a bit difficult to track the conversation above, but a feature of separating the component into additive contributions would be really helpful for analyses. My attempts to do this (by zeroing out the variances in the model before sampling, trying to mask different kernels) definitely were unable to solve this and recover the component contributions.


PT-6909 Apply matcher aliases to tables on config load

The last piece extracted from https://github.com/hellofresh/klepto/pull/110
Instead of passing matcher aliases everywhere and checking them - they are checked and replaced on config load once.
Dependend on https://github.com/hellofresh/klepto/pull/112 and has to be rebased after merge.

:+1:


Support HTTP/2 and QUIC

The current product supports HTTP/1.1 only.
We want to support HTTP/2 and QUIC (HTTP/3) for better pipelining.

Oh - this was easy - works out of the box (I was surprised too!)


Migrate via PS

@alkohli

Can you review this PR?
IMPORTANT: When this content is ready to merge, you must add #sign-off in a comment or the approval may get overlooked.

We'll review this after approval, but I already can see one issue (without reading the article): All new articles require a TOC entry in the same PR.
NOTE: this PR introduces a new article, and as such the commits will need to be moved to the private repo (after approval) for full review.
#label:"aq-pr-triaged"
@MicrosoftDocs/public-repo-pr-review-team

If this PR needs approval, or if the commits need to be pulled into the private repo, contact @ManikaDhiman. Thanks.

This is a draft working version - not ready for review


Sub-sub domain fix

From @phoenix-zhu 's request in #10....

to get a cert for a non-secondary level domain like 'a.b.c.com' because the code here(https://github.com/ocelotconsulting/node-letsencrypt-lambda/blob/master/src/aws/route53/getHostedZoneId.js#L8) only cuts the last 2 parts of the domain for zone name.

@phoenix-zhu ... I just pushed a632f88e, which I think should solve your issue. It requires a little more configuration, as I have no idea the limits at which a user might utilize domain levels with hosted zones (theoretically up to 127, but not sure who would do that). Please see the documentation on the differences in the config JSON, and let me know what you think. If this works for you, I'll close the issue. Thanks!

Thanks, @larryboymi it works fine.
I really appreciate all the update you've done for me. You are awesome.

No problem @phoenix-zhu ... thanks for your help in finding these outlying issues. Glad I could help.


Support Avro 1.10.0

Support Avro schema 1.10.0 (enum type and so ones ...)

Hey @M3lkior. Currently we use the https://github.com/mtth/avsc package.
According to the docs, it uses the 1.8.1 specs of avro.
We've just upgrade Javro to the latest version of avsc, hope it helps :)


Add support for more than one seeder path provided

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
0.0%

Covered Lines:
108

Relevant Lines:
108

💛  - Coveralls


fix alt text and figure out why images are missing

Visit https://docs.nimbella.com/deploy-from-github and you'll see this at the end bottom of the page:

There are three references to this the code base, as alt-text for images.
https://github.com/nimbella/docs/search?q=Docusaurus+with+Keytar
https://github.com/nimbella/docs/blob/877fecaa36d480b281066a8ed23d055fd37879f8/docs/key-value-storage.md#L36
https://github.com/nimbella/docs/blob/5bdc4b88c42ed39c94807dd7ed25dc466df4613f/docs/project-creating-your-own.md#L21
https://github.com/nimbella/docs/blob/5bdc4b88c42ed39c94807dd7ed25dc466df4613f/docs/deploy-from-github.md#L39

Fixed by https://github.com/nimbella/docs/commit/28e97404e87801e7c47f1ad98240985bf5737cc2 https://github.com/nimbella/docs/commit/51c94af6e7639b4978ad08bac6c33083eeb8f592 and https://github.com/nimbella/docs/commit/0038e07fd4a96f4c993bedec2b64dd5a4164f3ec.


Use interpolate option in Sea Level example

The setting of imageSmoothingEnabled in a layer prerender event pre-dates the imageSmoothing/interpolate option which would make the example simpler and consistent with other examples which now use that option.

Thanks, @mike-000


Add details on MACRO calibration from v2017 annex of MESSAGE-GLOBIOM documentation

Details on the implementation of MACRO, its parameterization and the calibration procedure can be found in the Annex. This information is going to be removed from the 2020 release of the MESSAGE-GLOBIOM documentation.

Hi is it possible for me to access the Annex now? The document requires permission to access. Or do I need to wait for an official release? Thank you.

@aaronliuqy the additional information that should be transferred to the MACRO documentation as part of MESSAGEix is still part of the earlier 2017 release of the MESSAGEix-GLOBIOM documentation. We are currently in the process of consolidating the MESSAGEix and MESSAGEix-GLOBIOM documentation which includes moving around some parts.

@volker-krey Thank you for the explanation. I thought the annex might contain the (potential) tutorial mentioned in thread #316.  It will be helpful if there is a tutorial for linking MESSAGE and Macro.

No one was assigned to do this by the feature freeze for v3.2.0, so we postpone to v3.3 (or later).

No one was assigned to do this by the feature freeze for v3.2.0, so we postpone to v3.3 (or later).

No one was assigned to do this by the feature freeze for v3.3.0, so we postpone to v3.4 (or later).

No one was assigned to do this for v3.4.0, so we postpone to v3.5 (or later).

No one was assigned to do this by the feature freeze for v3.5.0, so we postpone to v3.6 (or later).

Removing the milestone entirely. It can be re-added if/when the team decides to schedule/assign the work.


wrong titlebar in PDF reader

When I open the tutorial in my PDF reader, the title bar shows the old title, "Dependent Types in Lean". I can't figure out where it is getting that from -- the title is correct in all the .org files. What is going on?

It comes from https://github.com/leanprover/tutorial/blob/244afa00a09b705d0d0dd7cb1e9a7cc91630c919/header/header.tex#L38
I'll fix it tonight. Thanks for the report!


Switch to -print-resource-dir (drop support for system clang 4.0)

We don't support downloading Clang 4.0 anymore and I think -print-resource-dir might give better diagnostics to solve #550. This implies using Clang 4.0 from system won't be possible anymore.

Can you update the documentation as well?  Currently says "Clang 3.4 or higher"

Did you pass SYSTEM_CLANG to cmake at any point? Then the system has to provide libclang and thus clang will have to be 5.0 or higher

Yes, I'm using system libclang.  I'm having trouble understanding, you said:

only the libclang we download needs to be 5 or higher.

and

Did you pass SYSTEM_CLANG to cmake at any point? Then the system has to provide libclang and thus clang will have to be 5.0 or higher

If both system clang and downloaded clang need to be version 5 and above, which compilation method is compatible with

Compiling with Clang 3.4 still works

thanks!

There's two separate concerns here.

The compiler used to compile cquery. This can MSVC/GCC/Clang (with C++14 support). Clang 3.4 fits this case.
The libclang version that cquery links against. libclang is bundled with the Clang compiler. We only support libclang from Clang 5 and above. By passing SYSTEM_CLANG=ON to cmake you tell cmake to use the libclang from your system (which in your case is Clang 3.4 which isn't supported).

Since you don't have a supported version of libclang installed on your system, you have to use the downloaded Clang. You can however still use the Clang compiler installed on your system to compile cquery. The cmake command then becomes: cmake -DCMAKE_CXX_COMPILER=clang++ .. (SYSTEM_CLANG is off by default). CMake will then download Clang 6 and link cquery against the libclang bundled with that downloaded Clang while using the installed Clang to actually do the compilation


unregister flot.axislabels

please unregister package flot.axislabels.
i am receiving this error
EUNKNOWN Can only unregister packages hosted on GitHub.com
package is hosted at bitbucket.org
This was just a test and was hoping to unregister but I couldn't. Thanks

Unregistered flot.axislabels. Thank you for using Bower :bird:


DotNet Core - Publish - Deletes output folder causing Publish Artifact task to fail

Environment

Server - VSTS

Agent - Hosted 2017

Issue Description
Currently attempting to use the new DotNet core build template for a DotNet Core 2.1.0 WebAPI project. It appears that the default template causes issues in how it zips the output up and subsequently deletes the output folder. This causes issues with the default template in the downstream task of Publish Artefact as it can no longer find the location of the artefact itself.
I would not have expected this to be default behaviour for users attempting to use the template out of the box.
Workaround
Able to workaround by modifying the output directory in the Publish task to write to a subfolder. This then causes the zip to be placed 1 level up from this new subfolder, which is where the default location is for the subsequent Publish Artefact task.
ie. --configuration $(BuildConfiguration) --output $(build.artifactstagingdirectory)\artifact
Error logs

2018-06-07T02:24:17.8506421Z ##[debug]Zip Source: D:\a\1\a
2018-06-07T02:24:17.8508196Z ##[debug]Zip arguments: Source: D:\a\1\a , target: D:\a\1\a.zip
2018-06-07T02:24:18.0627450Z ##[debug]Successfully created archive D:\a\1\a.zip
2018-06-07T02:24:18.0628182Z ##[debug]rm -rf D:\a\1\a
2018-06-07T02:24:18.0629813Z ##[debug]removing directory

2018-06-07T02:24:18.3032197Z ##[debug]PathtoPublish=D:\a\1\a
2018-06-07T02:24:18.3032581Z ##[debug]check path : D:\a\1\a
2018-06-07T02:24:18.3051731Z ##[debug]task result: Failed
2018-06-07T02:24:18.3052522Z ##[error]Publishing build artifacts failed with an error: Not found PathtoPublish: D:\a\1\a

Are you using the template without any change? Could it be that you have un-set "Add project name to publish path" field. I am not able to repro this with the default template. If possible, can you share the complete debug logs (here or send them to RM_Customer_queries@microsoft.com).

The only thing I've unchecked from the default template is the Publish Web Projects' field (WebAPI project). I can confirm the Add project name to publish path` is still set.
I've emailed through the full logs to the above email address :)

I looked at the logs, and I think I understand the issue. This is when you pass "Path to project(s)" as empty - then we are not able to publish individual projects to separate sub-folders. You can use the same pattern as you have in other tasks "**/*.csproj" or the path to the project that you want to publish and then things should work fine.
The current work-around you have to put an artifacts sub-directory is not the optimal solution, since if you have multiple webApi projects - their published output will conflict with each other.
In the meantime, I will look at how best to fix this in the task code.

This issue has been fixed, will be rolled out in a few weeks! :)


[SPARK-45250][core] Support stage level task resource profile for yarn cluster when dynamic allocation disabled

What changes were proposed in this pull request?
This PR is a follow-up of https://github.com/apache/spark/pull/37268 which supports stage level task resource profile for standalone cluster when dynamic allocation disabled. This PR enables stage-level task resource profile for yarn cluster.
Why are the changes needed?
Users who work on spark ML/DL cases running on Yarn would expect stage-level task resource profile feature.
Does this PR introduce any user-facing change?
No
How was this patch tested?
The current tests of https://github.com/apache/spark/pull/37268 can also cover this PR since both yarn and standalone cluster share the same TaskSchedulerImpl class which implements this feature. Apart from that, modifying the existing test to cover yarn cluster. Apart from that, I also performed some manual tests which have been updated in the comments.
Was this patch authored or co-authored using generative AI tooling?
No

Hi @ivoson, I believe this PR is a follow-up of your previous PR https://github.com/apache/spark/pull/37268 since both yarn and standalone cluster share the same TaskSchedulerImpl. But I still would like to know what's your concern about this PR. Thx.

Please update description to say what this PR does. Specifically this does introduce user facing changes because there is now a new feature available to users of yarn.
Was this tested on a real cluster, what all tests were run other then unit tests?
You also need to update the documentation: docs/configuration.md, docs/running-on-yarn.md

Put it in draft mode until Kubernetes support is available. @tgravescs thx for the review.

Hi @ivoson, I believe this PR is a follow-up of your previous PR #37268 since both yarn and standalone cluster share the same TaskSchedulerImpl. But I still would like to know what's your concern about this PR. Thx.

Thanks @wbo4958 for ping me and work on this. No concerns about adding the support for yarn cluster. Please feel free to go ahead.

Manual tests
Due to the challenges of conducting yarn application tests within Spark unit tests, I took the initiative to manually perform several tests on our internal Yarn cluster.
With dynamic allocation disabled.
spark-shell --master yarn --num-executors=1 --executor-cores=4 --conf spark.task.cpus=1 \
   --conf spark.dynamicAllocation.enabled=false

The above command requires 1 executor with 4 CPU cores, and the default task.cpus = 1, so the default tasks parallelism is 4 at a time.

task.cores=1

Test code:
import org.apache.spark.resource.{ResourceProfileBuilder, TaskResourceRequests}

val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)

val treqs = new TaskResourceRequests().cpus(1)
val rp = new ResourceProfileBuilder().require(treqs).build

rdd1 = rdd1.withResources(rp)
rdd1.collect()

When the required task.cpus=1, executor.cores=4 (No executor resource specified, use the default one), there will be 4 tasks running for rp.
The entire Spark application consists of a single Spark job that will be divided into two stages. The first shuffle stage comprises four tasks, all of which will be executed simultaneously.

And the second ResultStage comprises 3 tasks, and all of which will be executed simultaneously since the required task.cpus is  1.

task.cores=2

Test code,
import org.apache.spark.resource.{ResourceProfileBuilder, TaskResourceRequests}

val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)

val treqs = new TaskResourceRequests().cpus(2)
val rp = new ResourceProfileBuilder().require(treqs).build

rdd1 = rdd1.withResources(rp)
rdd1.collect()

When the required task.cpus=2, executor.cores=4 (No executor resource specified, use the default one), there will be 2 tasks running for rp.
The first shuffle stage behaves the same as the first one.
The second ResultStage comprises 3 tasks, so the first 2 tasks will be running at a time, and then execute the last task.

task.cores=3

Test code,
import org.apache.spark.resource.{ResourceProfileBuilder, TaskResourceRequests}

val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)

val treqs = new TaskResourceRequests().cpus(3)
val rp = new ResourceProfileBuilder().require(treqs).build

rdd1 = rdd1.withResources(rp)
rdd1.collect()

When the required task.cpus=3, executor.cores=4 (No executor resource specified, use the default one), there will be 1 task running for rp.
The first shuffle stage behaves the same as the first one.
The second ResultStage comprises 3 tasks, all of which will be running serially.

task.cores=5

import org.apache.spark.resource.{ResourceProfileBuilder, TaskResourceRequests}

val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)
val treqs = new TaskResourceRequests().cpus(5)
val rp = new ResourceProfileBuilder().require(treqs).build

rdd1 = rdd1.withResources(rp)

exception happened.
scala> rdd1 = rdd1.withResources(rp)
org.apache.spark.SparkException: The number of cores per executor (=4) has to be >= the number of cpus per task = 5.
  at org.apache.spark.resource.ResourceUtils$.validateTaskCpusLargeEnough(ResourceUtils.scala:412)
  at org.apache.spark.resource.ResourceProfile.calculateTasksAndLimitingResource(ResourceProfile.scala:182)
  at org.apache.spark.resource.ResourceProfile.$anonfun$limitingResource$1(ResourceProfile.scala:152)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.resource.ResourceProfile.limitingResource(ResourceProfile.scala:151)
  at org.apache.spark.resource.ResourceProfileManager.addResourceProfile(ResourceProfileManager.scala:141)
  at org.apache.spark.rdd.RDD.withResources(RDD.scala:1829)
  ... 50 elided

scala>

With dynamic allocation enabled.
spark-shell --master yarn --num-executors=1 --executor-cores=4 --conf spark.task.cpus=1 \
  --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.maxExecutors=1\

The above command enables the dynamic allocation and the max executors required is set to 1 in order to test.
TaskResourceProfile without any specific executor request information
Test code,
import org.apache.spark.resource.{ResourceProfileBuilder, TaskResourceRequests}

val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)

val treqs = new TaskResourceRequests().cpus(3)
val rp = new ResourceProfileBuilder().require(treqs).build

rdd1 = rdd1.withResources(rp)
rdd1.collect()

The rp refers to the TaskResourceProfile without any specific executor request information, thus the executor information will utilize the default values from Default ResourceProfile (executor.cores=4).
The above code will require an extra executor which will have the same executor.cores/memory as the default ResourceProfile.

Different executor request information
import org.apache.spark.resource.{ExecutorResourceRequests, ResourceProfileBuilder, TaskResourceRequests}

val rdd = sc.range(0, 100, 1, 4)
var rdd1 = rdd.repartition(3)

val ereqs = new ExecutorResourceRequests().cores(6);
val treqs = new TaskResourceRequests().cpus(5)

val rp = new ResourceProfileBuilder().require(ereqs).require(treqs).build

rdd1 = rdd1.withResources(rp)
rdd1.collect()

You still need to update the documentation like I mentioned here: https://github.com/apache/spark/pull/43030#issuecomment-1735586052
Also need to look at the build failure, doesn't look like this code so maybe something with setup in your repo.

Thx @tgravescs, I reran the build/tests, but they still kept failing. Let me rerun them again.

@HyukjinKwon  would you have any idea on the build failure? Looks like env is just not setup properly but I've never seen that before.

Hi @tgravescs, Finally, I rebased this PR and forced-updated my PR, now the CI got passed. Thx

Merged to master.
Thanks for working on this @wbo4958 !
Thanks for the reviews @tgravescs and @ivoson :-)

I did not backport it given it was an improvement, but don't have objections as such - as you said, it is low risk

thanks, I merged back into 3.5 branch (3.5.1) as well.


Volume with names longer than 27 characters

How should this be handled?

Could be interesting to try and figure out how CFURLCreateBookmarkDataFromFile does it. It seems like they produce a different kind of alias format. Strange :S

Hello, is there any news about this problem ? When you plan release a fix ? Could you be more specific and explain "I did a quick try and it turns out that truncating the volume name in the alias does indeed work." how can I do this by myself ?

The volume name is stored twice in the alias and only the first occurrence have a limit of 27 characters. Writing anything into the first one and then the correct information into the second one works around the problem.
The code thats add the second occurrence is in create.js, line 118-132.
The current code is throwing at encode.js, line 40-44. Only using the 27 first characters from info.volume.name at this particular point works around the problem.

There should however be a better solution, since CFURLCreateBookmarkDataFromFile produces aliases that works. We should figure out how they work and use that approach.

What is the reason for such limitation?
I'm using electron-forge to build my electron app to a macOS image and I'm hitting this AssertionError [ERR_ASSERTION]: Volume name is not longer than 27 chars because it seems to be trying to create a .dmg file with a name longer than 27 characters.
After a few Google research, I didn't find anything about this 27 characters limitations 🤔
For example, https://www.quora.com/What-is-the-maximum-length-of-a-filename-and-pathname-in-macOS

Very old ticket here, but still I am limited to 27 chars for a DMG (producing an Electron app with Forge).  Is this still a limitation?

Hey @busterbogheart, we patched out this assertion locally with patch-package and the generated DMG works fine in production.


Set up task allocation

What is the problem you were trying to solve?
Set up folders and framework for initial task allocation
What is your solution and why did you solve this problem the way you did?
Provide any additional notes on testing this functionality:
Screenshots / gifs of working solution:

Nice!


[updater] fix bootstrap

What does this PR do?

remove bootstrap dependency on datadog.yaml: keep the pattern of working package installation when the config file is not set
pin agent versions that won't get deleted for default install

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

/merge


feat: improve URI handling by basing it on the libcoap type

This PR rewrites the CoapUri type to work directly with an underlying coap_uri_t object and use the libcoap-provided string parsing functions.
Additionally the url crate is now an optional dependency, and the CoapUri type now implements FromStr, which allows way easier construction of URIs.
Before (using url crate):
let uri = CoapUri::try_from_url(Url::parse("coap://example.com:4711/foo/bar?answer=42")?)?;

Before (without url crate):
let uri = CoapUri::new(
        CoapUriScheme::Coap,
        Some(CoapUriHost::Name("example.com".to_string())),
        Some(4711),
        Some(vec!["foo".to_string(), "bar".to_string()]),
        Some(vec!["answer=42".to_string()]),,
    );

Now:
let uri: CoapUri = "coap://example.com:4711/foo/bar?answer=42".parse()?;

Apparently there seems to be some issue with doctests and the unstable --ensure-time cargo flag.
For now, I have disabled these unstable flags and changed the toolchain the tests are run on to stable Rust.


ABA issue with freeing and allocation of command buffers and destruction of command pools in core validation layer

We have stumbled recently on a bug in core validation layer. The bug lies in the layer's DestroyCommandPool API entry function. Here is the function:
VKAPI_ATTR void VKAPI_CALL DestroyCommandPool(VkDevice device, VkCommandPool commandPool, const VkAllocationCallbacks *pAllocator) {
    layer_data *dev_data = GetLayerDataPtr(get_dispatch_key(device), layer_data_map);
    unique_lock_t lock(global_lock);
    bool skip = PreCallValidateDestroyCommandPool(dev_data, commandPool);
    if (!skip) {
        lock.unlock();
        dev_data->dispatch_table.DestroyCommandPool(device, commandPool, pAllocator);
        lock.lock();
        if (commandPool != VK_NULL_HANDLE) {
            PostCallRecordDestroyCommandPool(dev_data, commandPool);
        }
    }
}

The problem is that the command pool's state is erased after destroying the command pool in the next layer while unlocked. After this line executed
        dev_data->dispatch_table.DestroyCommandPool(device, commandPool, pAllocator);

any other thread can allocate any number of command buffers, and it might (and will, we did get it) happen that memory freed by the above line is reused, and thus it's state will reuse the existing entry in the layer's command buffer map . So when we get to
            PostCallRecordDestroyCommandPool(dev_data, commandPool, cp_state);

it will destroy the command buffer's state, leaving the created command buffer with no state, and later on, we hit an assert (or a crash) about the command buffer's state being null.
I do not know why clearing the layer's command pool data must be done after calling the next layer's function. Looking in what PostCallRecordDestroyCommandPool does, it does not seem to require that order. Then again I might miss something. On our end, I replaced the code inside if (!skip) for
    //...
    if (!skip) {
        if (commandPool != VK_NULL_HANDLE) {
            PostCallRecordDestroyCommandPool(dev_data, commandPool, cp_state);
        }
        lock.unlock();
        dev_data->dispatch_table.DestroyCommandPool(device, commandPool, pAllocator);
    }
    //...

and aside from the fact that the function now does not bare the right name (rename it as PreCallRecordDestroyCommandPool?), it solved our problem. It also follows a normal reverse-order destruction sequence.
P.S.: The unlock/lock pattern around the next layer's API call and doing the core validation's associated work (other than validating) after that call is common in the code. I did not spend time to see if any other similar ABA issue exist.

Thanks, @fduranleau-gl I agree with your assessment and proposed fix. I'll create a formal PR and credit you.

@tobine, he has an existing PR, which we've added to.

@mark-lunarg cool, yeah, I see it now #2518. Thanks, @fduranleau-gl

@fduranleau-gl  I'd like clarification on the usage pattern exposing the bug.
If you are saying that during the destruction of a command pool, that command buffers are being allocated from that same pool, that's a spec violation and an incomplete detection our the validation layers part to not report it.
Or... are you saying that when destroying one command pool, another is created and command buffers allocated from that pool are corrupted.  I'm trying to see the mechanism for failure and (absent unique objects) I could see a race where while deleting command pool with handle "A", another is created, given the same handle "A", meanwhile the destroy function is cleaning up the data from the destroyed version of "A".  (your fix, would be exactly correct for this failure mechanism)

@jzulauf-lunarg It's exactly the second case you mention that we hit.

@fduranleau-gl -- the case starting "Or..." or the case starting "I could also see" ?  (I should have numbered them :) )
Also, was the unique objects layer enabled?

Pushed changes to Master.  Thanks a bunch @fduranleau-gl.  I'll close the issue, but you guys can continue with the investigation in this issue.

@jzulauf-lunarg It would be the "Or...", or a situation like this:

Thread 1 destroys a command pool, from which a command buffer with handle value A has been created.
Thread 1 unlocks and executes the down-chain call.
Thread 2, before thread 1 regains the lock, creates a command buffer from another pool. That command buffer gets handle value A.
Thread 1 regains the lock, and removes layer state data for command buffer with handle value A from the map.
Thread 2 executes a command with its newly allocated command buffer, but it's layer state has been destroyed.

The enabled layer was VK_LAYER_LUNARG_standard_validation, so yes, but it is higher up in the call chain.

@jzulauf-lunarg, Vulkan command buffer objects are dispatchable.  Unique objects wraps only non-dispatchable objects.  So all is right in the world again.


Fix package name to eventing-contrib.

Fixes #372
`knative/eventing-sources` changed to `knative/eventing-contrib`

/lgtm

/lgtm

I bet tide is broken on this repo

yeah... 😬
On Sat 8. Jun 2019 at 05:03, Scott Nichols<EMAIL_ADDRESS>wrote:

I bet tide is broken on this repo
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/knative/eventing-contrib/pull/452?email_source=notifications&email_token=AABGPTUB2FL5IF7TV7ZIZEDPZMOP7A5CNFSM4HV3N7QKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXHL34A#issuecomment-500088304,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AABGPTQSZ22MQKD6R63Q4JDPZMOP7ANCNFSM4HV3N7QA
.

--
Sent from Gmail Mobile

/retest


[Q] How to parse this particular SCTE-35 message?

Hi,
I am trying to parse SCTE-35 message with hex data: fc303a00000000000000fff00506ffe3d359f80024022243554549040001257fc000001b77400e0c41594c30303030303030303120010101012c124463
I have the below function to get the parsed SCTE-35 message
SCTE35Message SCTE35Parser::parse(const uint8_t* packetData, size_t packetSize) {
    SCTE35Message scte35Message;

    // Create a section from the data pointer
    ts::BinaryTable binaryTable;
    ts::SectionPtr sectionPtr = ts::SectionPtr(new ts::Section(packetData, packetSize));
    binaryTable.addSection(sectionPtr);

    ts::SpliceInformationTable sit(duckContext, binaryTable);
    ts::TablesDisplay display(duckContext);

    ts::Descriptor descriptor(static_cast<const void*>(packetData), packetSize);

    scte35Message.isValid = sit.isValid();
    scte35Message.data = utils::byteToHex(packetData, packetSize);
    if (sit.isValid()) {
        // Extracting relevant details for scte message
        scte35Message.protocolVersion = sit.protocol_version;
        scte35Message.spliceCommandType = this->getSpliceCommandType(sit.splice_command_type);

        for (int i = 0; i < sit.descs.count(); i++) {
            const ts::SpliceSegmentationDescriptor ssd(duckContext, *sit.descs[i]);
            scte35Message.identifier = utils::decimalToHex(ssd.identifier);
            scte35Message.noRegionalBlackout = ssd.no_regional_blackout;
            scte35Message.segmentNum = ssd.segment_num;
            scte35Message.segmentationTypeId = utils::decimalToHex(ssd.segmentation_type_id);
            scte35Message.segmentationDescription = this->getSegmentationDescription(ssd.segmentation_type_id);
            scte35Message.segmentationEventId = utils::decimalToHex(ssd.segmentation_event_id);
            scte35Message.segmentationUpidType = utils::decimalToHex(ssd.segmentation_upid_type);
            scte35Message.programSegmentation = ssd.program_segmentation;
            scte35Message.segmentationEventCancel = ssd.segmentation_event_cancel;
            scte35Message.webDeliveryAllowed = ssd.web_delivery_allowed;
            scte35Message.upidData = utils::byteBlockToHex(ssd.segmentation_upid);
            scte35Message.adId = utils::hexToAscii(scte35Message.upidData);
        }

        if (sit.splice_command_type == ts::SPLICE_TIME_SIGNAL) {

        }
    } else {
        std::cerr << "Invalid SCTE-35 data" << std::endl;

    }
    return scte35Message;
}

And a display function to display the message
void SCTE35Parser::display(const uint8_t* packetData, size_t packetSize) {

    // Create a section from the data pointer
    ts::Section section(packetData, packetSize, ts::PID_NULL, ts::CRC32::CHECK);

    if (section.isValid()) {

        ts::TablesDisplay display(duckContext);
        // Display the section
        display.displaySection(section);
        std::cout << "\n";
    }
}

the displaySection function of ts::TablesDisplay  is displaying the message correctly
* unknown (0xFC), TID 0xFC (252)
Short section, size: 61 bytes
Protocol version: 0
Encryption: none
PTS adjustment: 0x000000000
CW index: 0x00 (0), tier: 0xFFF (4095)
Command type: 0x0000000000000006 (unknown), size: 5 bytes
Time: 0x1E3D359F8 (8,117,246,456, 90,191,627 ms)
- Descriptor 0: unknown (0x02, 2), 34 bytes
Identifier: 0x43554549 ("CUEI")
Segmentation event id: 0x04000125, cancel: 0
Program segmentation: 1, has duration: 1, not restricted: 0
Web delivery allowed: 0, no regional blackout: 0
Archive allowed: 0, device restrictions: 0
Segment duration: 1800000
Segmentation upid type: 0x000000000000000E (unknown)
Upid data (12 bytes):
    0000:  41 59 4C 30 30 30 30 30 30 30 30 31              AYL000000001
Segmentation type id: 0x0000000000000020 (unknown)
Segment number: 1, expected segments: 1
Extraneous 2 bytes:
 0000:  01 01                                            ..
CRC32: 0x2C124463 (OK)

But when I parse this using the parse function given above and print the SCTE35Message, I am getting an invalid message
 SCTE35Message {isValid: true, encodedData: 'fc303a00000000000000fff00506ffe3d359f80024022243554549040001257fc000001b77400e0c41594c30303030303030303120010101012c124463', protocolVersion: 0, spliceCommandType: 'TimeSignal', identifier: '0x43554549', noRegionalBlackout: true, segmentNum: 0, segmentationTypeId: '0x00', segmentationDescription: 'Unknown Segmentation Type ID', segmentationEventId: '0x00', segmentationUpidType: '0x00', programSegmentation: true, segmentationEventCancel: false, webDeliveryAllowed: true, upidData: '', adId:  }

Can anyone please help me on how to parse this correctly?

Hi, @arhankundu99
This is the same problem as #1456. Your splice section is globally correct but your segmentation descriptor is incorrect. It has 2 extraneous bytes. The display function does it best to display what is inside the section (including the extraneous bytes). But the parsing of the descriptor, as a C++ object, will fail because the binary data are invalid.
$ echo fc303a00000000000000fff00506ffe3d359f80024022243554549040001257fc000001b77400e0c41594c30303030303030303120010101012c124463 \
   | xxd -r -p | tstabdump -

* SCTE 35 Splice Information, TID 252 (0xFC)
  Short section, size: 61 bytes
  Protocol version: 0
  Encryption: none
  PTS adjustment: 0x000000000
  CW index: 0x00 (0), tier: 0xFFF (4095)
  Command type: 0x06 (TimeSignal), size: 5 bytes
  Time: 0x1E3D359F8 (8,117,246,456, 90,191,627 ms)
  - Descriptor 0: SCTE 35 Segmentation (0x02, 2), 34 bytes
    Identifier: 0x43554549 ("CUEI")
    Segmentation event id: 0x04000125, cancel: 0
    Program segmentation: 1, has duration: 1, not restricted: 0
    Web delivery allowed: 0, no regional blackout: 0
    Archive allowed: 0, device restrictions: 0
    Segment duration: 1800000
    Segmentation upid type: 0x0E (ADS Information)
    Upid data (12 bytes):
      0000:  41 59 4C 30 30 30 30 30 30 30 30 31              AYL000000001
    Segmentation type id: 0x20 (Chapter Start)
    Segment number: 1, expected segments: 1
    Extraneous 2 bytes:
    0000:  01 01                                            ..
  CRC32: 0x2C124463 (OK)

Additionally, there are several other issues.
First, you have probably not correctly installed the TSDuck library (make install or .rpm or .deb package file). We can see that in your display function. The structure of the section is correctly displayed but the name of the section and the descriptor, as well as the splice command type are set as "unknown". The names are in configuration files. If you did not properly install the library and its development environment, you won't see any meaningful name, and other weird behaviours may happen.
The following code is also invalid:
ts::Descriptor descriptor(static_cast<const void*>(packetData), packetSize);

Your binary data contain a section, not a descriptor. So, you cannot do anything with this variable. You don't use it anyway, so I assume it was a former incorrect attempt.
After this code, you use ssd without checking that it is valid (ssd.isValid()).
const ts::SpliceSegmentationDescriptor ssd(duckContext, *sit.descs[i]);

A descriptor can be anything, not necessarily a splice segmentation descriptor. So, you have to check that the descriptor was properly parsed as the expected type.
Furthermore, in your case, event though your descriptor is a segmentation one, it is incorrect because of the extraneous bytes. So, the parsing fails. You would see it by checking ssd.isValid(). Therefore, the variable ssd does not contain any correct descriptor (probably the result of the default constructor).

But using the code I mentioned above, the SCTE message is getting parsed correctly for valid hex message.
Valid hex message: 'fc303300000000000000fff00506fe1099fde8001d021b43554549040002377f800e0c41594c303030303030303132210101627c1224
The display function prints the below message
 Short section, size: 54 bytes
Protocol version: 0
Encryption: none
PTS adjustment: 0x000000000
Command type: 0x0000000000000006 (unknown), size: 5 bytes
Time: 0x01099FDE8 (278,527,464, 3,094,749 ms)
- Descriptor 0: unknown (0x02, 2), 27 bytes
Identifier: 0x43554549 ("CUEI")
Segmentation event id: 0x04000237, cancel: 0
Program segmentation: 1, has duration: 0, not restricted: 0
Web delivery allowed: 0, no regional blackout: 0
Archive allowed: 0, device restrictions: 0
Segmentation upid type: 0x000000000000000E (unknown)
Upid data (12 bytes):
    0000:  41 59 4C 30 30 30 30 30 30 30 31 32              AYL000000012
Segmentation type id: 0x0000000000000021 (unknown)
Segment number: 1, expected segments: 1
CRC32: 0x627C1224 (OK)

And the SCTEMessage I get from parse function is this
SCTE35Message {isValid: true, encodedData: 'fc303300000000000000fff00506fe1099fde8001d021b43554549040002377f800e0c41594c303030303030303132210101627c1224', protocolVersion: 0, spliceCommandType: 'TimeSignal', identifier: '0x43554549', noRegionalBlackout: false, segmentNum: 1, segmentationTypeId: '0x21', segmentationDescription: 'Chapter End', segmentationEventId: '0x4000237', segmentationUpidType: '0x0e', programSegmentation: true, segmentationEventCancel: false, webDeliveryAllowed: false, upidData: '41594c303030303030303132', adId: AYL000000012 }

But using the code I mentioned above, the SCTE message is getting parsed correctly for valid hex message.

Precisely because the second message is correct and the first one incorrect, one is correctly parsed and the other one is not.
Using the trick echo hexa-text | xxd -r -p | tstabdump, you can easily see this. Because of the extraneous two bytes in the segmentation descriptor, the parsing fails.
I don't understand what is your problem exactly. Failing to parse an incorrect message and succeeding in parsing a correct message is exactly what is expected.

No sorry for the confusion, what I meant to say is that even though the hex message was invalid, the display function was still able to print the message.
I was wondering if I can somehow parse that data in my struct using splice information table class.

The display and deserialize functions are two different kinds of "parse".
The display function tries its best to display was can be parsed and give some clue on what is incorrect.  This is debug tool.
The deserialization function, on the other hand, cannot report incorrect data. There would be no way to check which field is correct or not. So, deserialization must reject incorrect data. No alternative.

Makes sense, thank you so much for taking time out!


[GPT-728] Full compilation via Template Unified Compiler

Passed verbose to it. Thanks for the suggestion.


[Depends on fog/fog#3384] Collect Hosts inventory for OpenstackInfra provider

Collecting Hosts inventory and hardware specs for OpenstackInfra
provider.
DependsOn: https://github.com/fog/fog/pull/3384

How do I run the miq-bot locally?

Would like to see if we can get VCR tests in here like we have for the other refreshers.  We might have to set up an environment to do so.

Yes, looking at tests now.

I am missing some docs on how to generate vcr cassette, could you point me to some?

tests added

Fog upstream is merged...  Waiting on Fog release.

Dependent fog patch has been released

yaay, CI finally passing :-)

@fryguy, I'm good with this after @Ladas scrubs the VCR cassette.
What do you think?

@Fryguy @blomquisg The credentials are fake for anybody except me, since it's all in my localhost. So I would consider them scrubbed. :-) If I will put there another random credentials, I will have to do a lot of rebasing. :-)

@Ladas, that's fine for now.  I'll merge this and you can get rebasing on the several dependent PRs.  We can revisit the vcr scrubbing after we get a non-localhost test environment.


local image matching returning duplicates and distant "matches"

the light blue dots are the sampled points, the white camera icons are the locations of the images (the screenshot is of JOSM, a program that by chance makes it drag-and-drop to display the location of geotagged images on a map). the OSM roads extract i used, and therefore also the sampled points, extended beyond the area for which I captured imagery. the orange circle is around the location of image with filename GS010002_0_002510.jpg

(for this screenshot i was using QGIS, a FOSS desktop GIS software, which is why it looks different) the yellow dots all have data/images/ParkView/GS010002_0_002510.jpg as the assigned image. we shouldn't have duplicates like this and they are all some distance from the actual location of that image.

part of the solution may be to configure a sampling distance as part of #38 that is used in the create_points process (instead of passing the  --mini-dist argument to set a distance in meters between interpolated points. then we can use that value in the imagery fetch step. we could divide it by 2 and use that distance as the buffer distance to search for matches. that is, if using the suggested distance of 20m between points, we would only consider images within 10 meters of a given point for a match. if we do this, i think there is still something off with the matching because of the duplicates we're seeing and the distance between the matches and the actual image location.

i ran this after @dragonejt made some changes and it now returns as expected (only the green dots that match where imagery is present) and with no duplicate images. we should still take a close look at the image matching/selection process as part of #81


Error during installation

[root@unit237 html]# ls
php-crawler
[root@unit237 html]# composer create-project hedii/php-crawler php-crawler
[InvalidArgumentException]
Could not find package hedii/php-crawler with stability stable.
create-project [-s|--stability STABILITY] [--prefer-source] [--prefer-dist] [--repository REPOSITORY] [--repository-url REPOSITORY-URL] [--dev] [--no-dev] [--no-plugins] [--no-custom-installers] [--no-scripts] [--no-progress] [--keep-vcs] [--no-install] [--ignore-platform-reqs] [--] [] [] []
What am I doing wrong?

try this and let me know if it is ok:
composer self-update && composer create-project hedii/php-crawler php-crawler
The package has 'stability stable', maybe composer is trying to get an old version. I don't have this problem with an up to date composer.

This is the output:
`You are already using composer version 0a8528c5c7de43eca15237add666b4f8219aedc4.
[InvalidArgumentException]
Could not find package hedii/php-crawler with stability stable.
create-project [-s|--stability STABILITY] [--prefer-source] [--prefer-dist] [--repository REPOSITORY] [--repository-url REPOSITORY-URL] [--dev] [--no-dev] [--no-plugins] [--no-custom-installers] [--no-scripts] [--no-progress] [--keep-vcs] [--no-install] [--ignore-platform-reqs] [--] [] [] []
`

What is your php version?
The required version is ">=5.5.9" (same as laravel)

PHP 5.3.3 (cli) (built: Feb  9 2016 10:36:17)
I will update it asap.

yes, update it :)


docs(fab): document FAB

rendered: https://github.com/material-components/material-web/blob/fab-docs/docs/components/fab.md

Sorry this is external but ping for review

Reviewing now - sorry for missing this!

my nit: can we import and do the review in critique for tracking? 😅

will import to critique rn


Can't record to v4l loopback

Attempting to record to the v4l2 loopback I get the following error:
❯ ./build/wf-recorder --muxer=v4l2 --codec=rawvideo --file=/dev/video5
selected region 0 0 0 0
Choosing pixel format bgr0
Output #0, video4linux2,v4l2, to '/dev/video5':
    Stream #0:0: Unknown: none (rawvideo)
avio_open failed

It could be a permissions problem, is your user part of the video group?

@soreau Yep:
❯ groups
wheel video kvm input lpadmin systemd-journal systemd-coredump portage plugdev lxd lovesegfault

Also you need to have v4l2loopback kernel module loaded which will create a device node /dev/video#.

@soreau Got it to work after reloading the module!

Hello, i am facing this same issue, can you please let me know how to reload that module, thanks


Support TargetRubyVersion 3.4 (experimental)

RuboCop AST 1.31 supports Ruby 3.4.
https://github.com/rubocop/rubocop-ast/pull/276
This PR supports TargetRubyVersion 3.4.

Before submitting the PR make sure the following are checked:

[x] The PR relates to only one subject with a clear title and description in grammatically correct, complete sentences.
[x] Wrote good commit messages.
[ ] Commit message starts with [Fix #issue-number] (if the related issue exists).
[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Ran bundle exec rake default. It executes all tests and runs RuboCop on its own code.
[x] Added an entry (file) to the changelog folder named {change_type}_{change_description}.md if the new code introduces user-observable changes. See changelog entry format for details.

🚀


Use new Search component from the Design system

Description

The Design system has created a new Search component, with better features and accessibility than the one we currently use (from the Altinn design system). We should thus switch to using this one wherever we use search
One thing we need to determine is if we should continue using a "hot reload" strategy for our search or use the new "submit" feature, that the new component facilitates.

Different strategies may be used for different places.
Conclusion
We will use the search field without button as we do today.
This is because fast feedback is nice for the user.
To limit the number of search calls, however, we will implement a debounce function to be used on the search field.
This way we can delay reaction to change in the search query by a few milliseconds.
Places to change:

API delegation:

Org search
API search

Single rights:

Service search

Additional Information
No response

Testet ok


Fix foreign key row reporting and empty table checks

Fix foreign key row reporting
My previous PR added a 'row' to the FK error, but it turned out to be wrong in many cases as it was actually a count of how many unique FK refs there were: this version generates a separate error record per FK violation and thus gives better feedback.  To do this is keeps a list of all rows which use a foreign key, which will use slightly more memory than before, but it also uses a hash to store the known values which should be somewhat quicker.
Run FK checks even if a table is empty
Previously, if any tables were empty (eg: a header but 0 rows) they wouldn't be marked as "valid" internally, so FK checks would not run. This PR fixes that by marking an empty table with a valid header as valid.  This was originally PR #190 but these issues are entangled so I'm closing that one and submitting this merged PR.

Ummm, Travis CI doesn't seem to be working again ...

Hmmm... That's weird. I can't replicate this locally. Will try and dig in to see what's going on...

Closing this; The :master branch in the remote repo has other changes beyond the original PR.


DATA-3594 Bump up StackDriver exporter TraceSpansBufferMaxBytes

Will hopefully resolve the following logs: OpenCensus Stackdriver exporter: failed to upload 10606 spans: buffer full

LGTM! Hopefully the corresponding change is app is effective :)