Detailed and Summary 24-Hour Reports failing on empty counts

What Happened
Some counts are "empty", in that they record no vehicles at all.  Detailed and Summary 24-Hour Reports fail to load on such counts.
What Should Happen
These reports should load successfully, and show that there are no counts.
To Reproduce
Steps to reproduce the bug:

Go to view/location/1/14659630/reports/ATR_SPEED_VOLUME in MOVE;
Click on '24-HOUR GRAPHICAL REPORT' or '24-HOUR SUMMARY REPORT' at top.

The report never loads; instead, the spinner continues spinning, and the backend issues an HTTP 500 error.
Screenshots
24-Hour Detailed Report failing to load:

To identify such a count:
-- find COUNT_INFO_ID values
select * from (select "COUNT_INFO_ID", sum("COUNT") as total from "TRAFFIC"."CNT_DET" group by "COUNT_INFO_ID") t where t.total = 0;

-- find count_group_id values
select * from counts.counts_multiday_runs where "COUNT_INFO_ID" = 1409307;

-- get centreline
select * from counts.studies where count_group_id = 1409307;

-- now visit /view/location/1/{centreline_id}/reports/ATR_SPEED_VOLUME

Can't repro in latest MOVE on an empty speed / volume ATR - all 4 relevant report types seem to work here.  Closing as fixed.

New repro courtesy of @aharpalaniTO : 24-hour detailed and summary reports fail at https://move.intra.dev-toronto.ca/view/location/1/1147135/reports/RESCU; this might be specific to particular study types?

Stacktraces:
Debug: internal, implementation, error
    TypeError: Cannot read property 't' of undefined
    at Function.transformCountData (/home/ec2-user/flashcrow/lib/reports/ReportCountSummary24hDetailed.js:21:36)
    at /home/ec2-user/flashcrow/lib/reports/ReportCountSummary24hDetailed.js:44:60
    at Array.map (<anonymous>)
    at ReportCountSummary24hDetailed.transformData (/home/ec2-user/flashcrow/lib/reports/ReportCountSummary24hDetailed.js:38:19)
    at ReportCountSummary24hDetailed.generate (/home/ec2-user/flashcrow/lib/reports/ReportBase.js:122:34)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async handler (/home/ec2-user/flashcrow/lib/controller/ReportController.js:106:26)
    at async exports.Manager.execute (/home/ec2-user/flashcrow/node_modules/@hapi/hapi/lib/toolkit.js:64:28)
    at async Object.internals.handler (/home/ec2-user/flashcrow/node_modules/@hapi/hapi/lib/handler.js:46:20)
    at async exports.execute (/home/ec2-user/flashcrow/node_modules/@hapi/hapi/lib/handler.js:31:20)

Debug: internal, implementation, error
    TypeError: Cannot read property 't' of undefined
    at Function.timeRange (/home/ec2-user/flashcrow/lib/reports/ReportCountSummary24h.js:52:36)
    at Function.peakSection (/home/ec2-user/flashcrow/lib/reports/ReportCountSummary24h.js:85:45)
    at transformCountData (/home/ec2-user/flashcrow/lib/reports/ReportCountSummary24h.js:116:42)
    at Array.map (<anonymous>)
    at /home/ec2-user/flashcrow/lib/reports/ReportCountSummary24h.js:188:35
    at Array.forEach (<anonymous>)
    at ReportCountSummary24h.transformData (/home/ec2-user/flashcrow/lib/reports/ReportCountSummary24h.js:176:34)
    at ReportCountSummary24h.generate (/home/ec2-user/flashcrow/lib/reports/ReportBase.js:122:34)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async handler (/home/ec2-user/flashcrow/lib/controller/ReportController.js:106:26)


CASMCMS-9045 - IMS clarify configuring remote build nodes.

Description
https://jira-pro.it.hpe.com:8443/browse/CASMCMS-9045
Clarify the documentation for the system admins on how to customize remote build nodes on their particular system.
Checklist

[X] If I added any command snippets, the steps they belong to follow the prompt conventions (see [example][1]).
[X] If I added a new directory, I also updated .github/CODEOWNERS with the corresponding team in [Cray-HPE][2].
[X] My commits or Pull-Request Title contain my JIRA information, or I do not have a JIRA.

/backport release/1.5


Add UNIX support on libevent

As requested by #2679

Modified Connection::read_callback_native to call specific data extraction function inNativeBufferEvent
Wrapped C specific function for data extraction from buffer (previous was okay but dirty and incompatible with UNIX use)
Added new class UnixConnectionListener wich extends ConnectionListener concept to UNIX sockets
Added new functions ConnectionFactory::accept_unix_connection and ConnectionFactory::bind_to_unix

:robot: add to whitelist

This PR is continued in #2726.


Update sprint-165-update.md

Applied bold to the "preview but don't run" in the description after the table

@AkshayaKulasekaran : Thanks for your contribution! The author(s) have been notified to review your proposed change.

Looks good; merging.
@gloridelmorales You can't use Markdown in an article metadata block. It renders as plain text:

I've removed it here. Thanks.


[cert-manager] get clusterIssuer reference in vcluster

As mentionned in code, host cluster clusterIssuers are not mapped by the cert-manager plugin.
Would it be possible to implement it?

It would technically be possible I believe, however that would require to have extended brace for the plugin, I will ask around for context and circle back to you. Also, we are moving to v2 of plugins, so if we are implementing it in the future that will be done there.

actually, we don't have plans to implement this, since that would require us to be able to list secrets in many namespaces, which I don't think is a good idea


Update files in vscode/src/vs/code to improve the copyright headers as mentioned in #96747.

This PR partially fixes #96747

This is a duplicate of https://github.com/microsoft/vscode/pull/104498.  It looks like that PR wasn't linked to the actual  issue, so you couldn't see it.  😕


uib-dropdown - ontoggle event not fired with multiple dropdowns on one page

When I have two dropdowns on the same page, both of which are using the on-toggle event, only one of them is called when i use the other dropdown to close the first.
In this plunker, open up your console and try this:
1, click dropdown1 to open it.  you'll see an event logged "dropdown1 open"
2. click dropdown1 to close it.  you'll see an event logged "dropdown1 closed"
3, click dropdown2 to open it.  you'll see an event logged "dropdown2 open"
4. click dropdown2 to close it.  you'll see an event logged "dropdown2 closed"
5, now click dropdown1 to open it.  you'll see an event logged "dropdown1 open"
4. click dropdown2 while dropdown1 is still open.  dropdown2 will open and dropdown1 will close but notice that the events only show the dropdown1 messages.  Since I'm doing other things in my on-toggle event, this is causing me a problem (and my users actually have to triple click the dropdown to get it to open)
http://plnkr.co/edit/SuaxVHxf84QG8KFFbvhY?p=preview

The problem manifests if at least two dropdowns are in the same container - in this case, both are using append-to-body, so they have the same parent element.
This would require us to use a multimap to fix this, as we would need to verify that there are no elements in the particular container before trying to remove the class.

@cleon26 did you devise a workaround for this issue? I'm experiencing the same problem

Sadly no.

@cleon26 I just ended up using ng-click on the same element


Update README.md

Add dependency package: python-mysqldb
Add install instruction: database migrate

+1, come to merge


clean: rename wrapper to eventer & wrap to event

Describe your changes
The wrapper field is renamed to eventer in the same way wrap is changed to event.
Issue ticket number and link
#18
Checklist before requesting a review

[X] I have performed a self-review of my code
[X] I have compiled and built the program.

Screenshots
Compiling Program

Running extractor

Running Logger

Also make sure to run cargo fmt before commiting. Otherwise the CI will complain :)
https://github.com/0xB10C/peer-observer/actions/runs/9961953175/job/27547091674?pr=24

Compiling Program

No need to include such screenshots. It's the task of a reviewer (with the help of CI) to verify the changes.

Oh and we should run https://github.com/0xB10C/peer-observer/blob/master/protobuf/gen-python.sh to regenerate the Python protobuf definitions. You can add the changes in a separate commit.

Okay I shall make the changes!!

Looks good! Can you change this up into two commits similar to this:

clean: rename wrapper to event_msg & wrap to event
update: protobuf python-types

Having the second commit split off allows me to run gen-python.sh myself and see if we get any differences.
Once that's done, I think this is ready for merge.

Surely!!!


Governing Board 2024 Security Goals

RFC to track Governing Board Dependability Pillar 2024 Security Goals

Zephyr Project shall facilitate product developers to be able to align with security requirements for products as required and defined by the various regulatory agencies.
No project blockers for ETSI EN 303-645 EU commission's Radio Equipment Directive
Security Committee / WG shall identify any relevant new security directives

### Tasks
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/50975
- [ ] https://github.com/zephyrproject-rtos/zephyr/issues/76700

@dleach02 - @kartben added the OpenSSF Scorecard to Zephyr's site last week.     We're at 5.3 - as some questions have interesting implications; and may not make sense for our project.   Something to discuss at next Security meeting.


Any way to fix this? Overlay appears with that "expand more details" checked.

I tried fiddling around with the JS in fixes.txt but I don't see any parameters relating to opacity

the shadow is due to the **.*.GameInfoShadow slement, where * are A-Z random letters that change with each update
901a44f2de1ee016c84cf3b43b89a1511a22d765


0.9.3 exception

Python 3.8.5 under Ubuntu 20.04 LTS
When I click Tools->Clear accumulator the program no longer works.
ERROR:2021-01-16 14:57:26,392:command_processor.py:384:joulescope_ui.command_processor:subscriber error for topic=Device/#state/play, value=True
Traceback (most recent call last):
  File "/home/x/python-virtual-environments/joulescope/lib/python3.8/site-packages/joulescope_ui/command_processor.py", line 382, in _subscriber_call
    fn(topic, value)
  File "/home/x/python-virtual-environments/joulescope/lib/python3.8/site-packages/joulescope_ui/main.py", line 1248, in _on_device_state_play
    self._device_stream_start()
  File "/home/x/python-virtual-environments/joulescope/lib/python3.8/site-packages/joulescope_ui/main.py", line 1211, in _device_stream_start
    self._accumulators_zero_last()
  File "/home/x/python-virtual-environments/joulescope/lib/python3.8/site-packages/joulescope_ui/main.py", line 764, in _accumulators_zero_last
    for z in self._accumulators['fields'].values():
TypeError: 'bool' object is not subscriptable

Hi @krbvroc1, and I apologize that you ran into this issue.  As a workaround for now, you can right-click on the Energy field in the Control Widget, then select Clear.
We are tracking it as #111, and it has already been fixed in the "develop" branch.  The fix will be in our next release.
I am going to close this issue as a duplicate, but please feel don't hesitate to comment or reopen it if you have questions.  Thanks again for taking the time to report this issue!

Hi @krbvroc1, and I apologize that you ran into this issue.  As a workaround for now, you can right-click on the Energy field in the Control Widget, then select Clear.
We are tracking it as #111, and it has already been fixed in the "develop" branch.  The fix will be in our next release.
I am going to close this issue as a duplicate, but please feel don't hesitate to comment or reopen it if you have questions.  Thanks again for taking the time to report this issue!

@mliberty1 - Thanks for the response. You mentioned a workaround of right clicking the Energy field and selecting Clear. Overnight I ran a test for 12 hours and this morning simply clicked the Energy field in the top right corner. I was expecting it to maybe toggle the units or something non-destructive, but instead it reset the value and I lost my in progress test. Is this expected behavior?

@mliberty1 - Thanks for the response. You mentioned a workaround of right clicking the Energy field and selecting Clear. Overnight I ran a test for 12 hours and this morning simply clicked the Energy field in the top right corner. I was expecting it to maybe toggle the units or something non-destructive, but instead it reset the value and I lost my in progress test. Is this expected behavior?

Yes, left-clicking on it will cause it to reset.  Right-clicking will give you options.  If you hit this accidentally, you can undo this Clear operation (and many other operations) using your platform-specific undo key combination.  On Windows, you use Ctrl-Z.

Yes, left-clicking on it will cause it to reset.  Right-clicking will give you options.  If you hit this accidentally, you can undo this Clear operation (and many other operations) using your platform-specific undo key combination.  On Windows, you use Ctrl-Z.

@mliberty1 Good to know. My suggestion would be to add a Clear to the right-click options so a left-click is not destructive. Thanks.

@mliberty1 Good to know. My suggestion would be to add a Clear to the right-click options so a left-click is not destructive. Thanks.


Refactor C++ allocators to be DRY and protected

As discussed in https://github.com/microsoft/mimalloc/pull/625:

Move some repeated code to base classes.
Protect the shared_ptr with the heap.

Thanks so much -- I refactored it a bit more and made it compile well with VS2022; hope it will work with most older compilers as well :-)

Cool! I've submitted a fix for comparions on GCC & some nice to have compile-time heap destruction strategy setting: https://github.com/microsoft/mimalloc/pull/653


Add fast post processing

Added fast post-processing from OpenVINO implementation. @druzhkov-paul Many thanks for making post-processing faster in times!

Great to see that you've found the code useful.

:accept:


Deleting tuples without known relation.

In my use-case I have groups, where an User can be of role member, super_member or owner. If I want to revoke his rights, I need to query every tuple containing his membership-type relation and then map it to a new tuple to be deleted. Could a feature be added in the future, where you can specify wildcard, or just not define relation at all to delete given user's tuples in given object?

@Nizio-Dev we're looking at improvements we'd like to introduce in a new major API version of OpenFGA. Part of those changes will likely include a Delete with filter endpoint where you can delete any tuples matching an arbitrary filter on any of the fields provided. I believe this would help you meet your ideal goals.
We're aware of our current API short-comings and have some roadmap plans to address them.

+1
An API version of the Tuple Management screen in the Dashboard UI would be very handy and I think would handle the OP and my own use-case (drop all tuples with user:bob regardless of object or relation).

We'll cover this scenario as part of this roadmap item https://github.com/openfga/roadmap/issues/34

Currently, the way to do that would be to call Read, and then follow that with a Write to delete all the returned tuples. Below is an example using the JS SDK
const { tuples } = await fgaClient.read({
  user: "user:81684243-9356-4421-8fbf-a4f8d36aa31b",
  object: "document:roadmap",
});

const { tuples } = await fgaClient.write({
  deletes: tuples, // make sure to chunk by 40 if the tuples are > 40
});

Or if this is a one of, and you do not want to use the SDK, you can use the CLI:
fga tuple read --user user:81684243-9356-4421-8fbf-a4f8d36aa31b --object document:roadmap --output-format=simple-json --max-pages=0 > tuples.json
fga tuple delete --file tuples.json


Linter and syntax formatter

Fixes #18
Adds pre-commit yaml file for automated listing. Also add documentation in ReadMe about pre-commit requirements

Should we run the linter/do cleanup as part of this PR?

regarding running the linter, the code in this branch is still in Python2, so I thought we could first merge it into the base with the Python3 coding and then open a new issue and run the linter then

I agree that it's probably a good idea to ignore certain flake8 error reports. The way to do that is in a config file, which you can read a bit more about here. There is a tox.ini file that's part of #31 but it might make sense to add a .flake8 file here so we don't get into too many merge conflicts.


move some extensions to ecosystem

Move the following modules to ecosystem, each extension as a single project.

dubbo-registry-redis

dubbo-remoting-grizzly

dubbo-remoting-p2p

dubbo-remoting-zookeeper

dubbo-remoting-mina

dubbo-rpc-memcached

dubbo-rpc-redis

dubbo-rpc-thrift

dubbo-rpc-webservice

dubbo-serialization-fst

The previous discussion thread is here<EMAIL_ADDRESS>

I sent a pull request #2330 for this issue, but how could I contribute to ecosystem ( https://github.com/dubbo), I cloud not create a dubbo-serialization-fst repository in the group.

As the mentors suggest us moving ecosystem to apache group, I think this task should be closed.
@ulinkwo Thanks for your work. Expecting see you continue to contribute to dubbo.


TEPHRA-152 Using ReferenceCounting for TransactionStateCache refresh …

…thread, so that it can be stopped

@gokulavasan Now that we have introduced class ReferenceCountedSupplier, it would be good to use this class in PruneUpperBoundWriterSupplier too. It will reduce code duplication.


Does not bind to new dom elements + jFiddle + solution

Created jFiddle to demonstrate: https://jsfiddle.net/1qr5av5g/
Possible solution: http://stackoverflow.com/questions/10920355/attaching-click-event-to-a-jquery-object-not-yet-added-to-the-dom
$(document).on();
$(document).on({
click: function () {},
mouseover: function () {},
mouseout: function () {}
}, '#element_id');

There is no function in JS to automatically bind something in new objects. Your example on events is not binding them, but it's just a technic called event delegation.
To bind new PS, you can simplify call again after it is added.
var $newDiv = $("<div>", {id: "foo", "class" : "container"});
$newDiv.appendTo(somewhere);
$newDiv.perfectScrollbar();

Please beware that jQuery support will be dropped in the next release.
Thanks.


Documentation using modules

Right now the starting guide (https:/router.vuejs.org/guide/) still uses a vanilla index.html approach. I think that it was easier back when vue-router started but now it isn't anymore and that maybe that page should be updated using .vue components and pointing to vue CLI and vite. It should maybe also point to Vue docs explaining Vue files.
What are your thoughts on this?
Do you have any other ideas on how to restructure this section of the docs?

https://www.vuemastery.com/blog/vue-router-a-tutorial-for-vue-3/
Found an example about how to use router in Vite.

starting a new vue3 + vite project, one thing that confused me was how to import the router in order to use the code from the getting started page.  ( import * as VueRouter from 'vue-router seems to do the trick. )
the previous version's installation page had an example import statement, and that was a nice help.

This was really painful, not having the imports needed for the examples on the router was quite confusing.

There have been several PRs to fix this rather obvious fault in the docs.. what do we need to do to get one in?
https://github.com/vuejs/router/pull/1951
https://github.com/vuejs/router/pull/1769
https://github.com/vuejs/router/pull/1323

@thedamon Most of them were actually incorrectly importing the router. It was better not to spread bad practices. Currently I need to focus on other stuff before getting back to the documentation, so feedback on difficult parts is welcome: https://github.com/vuejs/router/discussions/2068
BTW, https://github.com/vuejs/router/pull/1323 is the one that was the closest to what the page should look like but the author closed it probably because It's been 2 years

The examples have been updated by #2174.
I think this issue can be closed.


Facebook OAuth Internal facebook problem

Hi I'm having a problem that I just can't get rid of, it seems some facebook error, this is my code that I use to login and then get the profile info of the user:
    function facebookLogin() {
      return $q(function(resolve, reject) {
        $cordovaOauth.facebook('CLIENT_ID',  ['id', 'email', 'name', 'age_range', 'gender', 'user_photos', 'user_work_history', 'user_friends']).then(function(result) {
          $localStorage.facebookToken = result.access_token
          $http.get('https://graph.facebook.com/v2.8/me', { params: { access_token: $localStorage.accessToken, fields: 'id, name, email, gender, picture', format: "json" } }).then(function(profile) {
            alert(JSON.stringify(profile.data))
            resolve(true)
          }, function(err) {
            console.log(err)
          })
        }, function(err) {
            alert('Erro login: ' + JSON.stringify(err))
        })
      })
    }

I get this message error from facebook when it will render the login part and permissions for the user: "Something went wrong, We're working to get this fixed", don't know why. Sorry if it's not ng-cordova-oauth problem but stackoverflow didn't help me at all, thanks in advance.

Actually I was having problems with permissions in facebook which give a strange error, sorry for the inconvinience.


Sometimes, don't support using the TAB indent style format js code

VSCode Version: Code 1.7.1 (02611b40b24c9df2726ad8b33f5ef5f67ac30b44, 2016-11-03T13:53:14.822Z)
OS Version: Windows_NT ia32 10.0.14393
Extensions:

Extension
Author
Version

vscode-css-formatter
aeschli
0.0.2

vscode-caniuse
akamud
0.5.3

vscode-javascript-snippet-pack
akamud
0.1.5

vscode-theme-onedark
akamud
0.2.13

Bookmarks
alefragnani
0.10.0

project-manager
alefragnani
0.12.2

vscode-color
anseki
0.2.1

one-monokai
azemoh
0.1.15

tortoise-svn-for-vscode
cdsama
0.0.2

path-intellisense
christian-kohler
1.1.0

jshint
dbaeumer
0.10.15

vscode-instant-markdown
dbankier
0.1.0

githistory
donjayamanne
0.1.0

theme-dracula
dracula-theme
1.2.5

EditorConfig
EditorConfig
0.3.4

tslint
eg2
0.5.40

vscode-npm-script
eg2
0.0.21

tortoise-svn
fantasytyx
0.1.1

vscode-npm
fknop
3.3.0

docthis
joelday
0.3.5

Angular2
johnpapa
1.0.2

format-indent
Kasik96
1.3.0

ftp-sync
lukasz-wronski
0.3.1

debugger-for-chrome
msjsdiag
2.2.2

color-highlight
naumovs
1.2.1

promise-snippets
progre
0.2.3

vscode-icons
robertohuertasm
4.0.1

code-settings-sync
Shan
2.3.8

theme-cobalt0
sharksym
0.0.5

addDocComments
stevencl
0.0.8

js-complexity-analysis
TomiTurtiainen
1.0.4

Angular2
UVBrain
0.3.5

bootstrap-3-snippets
wcwhitehead
0.0.8

JavaScriptSnippets
xabikos
1.2.0

local-history
xyz
1.0.2

html-css-class-completion
Zignd
1.0.3

Steps to Reproduce:

@hxfdarling Can you provide some information on the issue you are facing? Am not able to understand much from just the title.
Please include repro steps, screenshots and sample code

This is my config
   "editor.mouseWheelScrollSensitivity": 1.8,
"editor.fontSize": 22,
"editor.renderWhitespace": "boundary",
"editor.mouseWheelZoom": true,
"editor.insertSpaces": false,
"editor.renderIndentGuides": false,
"editor.detectIndentation": false,

I have some code files can't according to the format TAB indent style,always use  space indentation
I can't reappear that when uninstall JS-CSS-HTML-formatter extensions after
Now I installed the JS-CSS-HTML-formatter extensions, solved this problem by this extensions

So installing  JS-CSS-HTML-formatter extension solved the problem?
Can you see the problem with all extensions disabled?


Automatically upload or update pdfs in zotero with pyzotero

Hi, thanks for the great package!
I'm looking for a way to automatically upload or update pdfs after I've compiled them in latex. Basically, I want to do the same than dragging a pdf file into a zotero collection in python.
Is this possible with pyzotero?
I have tried to upload a pdf with zot.attachment_both([(pdf_name, pdf_path)]) but while the file appears in my zotero, the attachment is not where zotero can find it (File not found error). What is the correct way to do this?
Also, how can I upload to a specific collection or subcollection?

pdf name is the name you want the attachment to have in Zotero, like "my_pdf_10" and path must be the full file path: /foo/bar/baz.pdf
In order to upload to a specific collection, you have to either attach to an existing item in that collection, or create an item in the collection and attach to that.

Is this the right syntax?
pdf_path = Path('~/mypdf.pdf').expanduser()
assert pdf_path.exists()
print(zot.attachment_both([('mypdf', str(pdf_path))]))

This way the file appears in zotero, but the attachment is not found.

I'm not sure what you mean by "file appears in Zotero, but the attachment is not found". The file is the attachment, and if you don't specify a parent ID, it will appear as a standalone file.
What happens when you double-click it?
What happens if you use attachment_simple(['/path/to/filename.pdf'])?
What happens if you use attachment_simple(['/path/to/filename.pdf'], parentid)?

When I double click it in the Zotero GUI under "My Library/Unfiled Items", I get the error message "File Not Found The attached file could not be found at the following path: /path/to/zotero/storage/ID/mypdf.pdf"
The same happens with attachment_simple(['/path/to/filename.pdf']).
I'm not sure how to use the parentid parameter. Is this the id of an item that already exists in the zotero database? If so, how can I find its id?
I have tried to upload another pdf with attachment_simple, but got the error:
pyzotero.zotero_errors.RequestEntityTooLarge:
Code: 413
URL: https://api.zotero.org/users/myid/items/id/file
Method: POST
Response: File would exceed quota (300.1 > 300)

Which is strange since the file is only 113 KB big, and I can upload it to zotero via the GUI.
Also, strangely the file still appears in the zotero GUI with an existing attachment, but the attachment is stored under the original path instead of the zotero storage under /path/to/zotero/storage/ after executing the command that results in the error.

The parent ID is the id of an existing item. You can retrieve items in a number of ways (see docs). "id" is a field in the returned dict(s).
Your entity too large error may be related to the total size of your Zotero library. If your storage quota is already full or nearly full, you won't be able to upload additional items.
I can't reproduce the original errors – both attachment methods are working for me both with and without parent IDs using the latest Pyzotero version on macOS.

Thanks for the info, I was able to add the attachment to an existing item with the id/key.
You were right, after emptying the bin, the RequestEntityTooLarge error disappeared.
In the dict returned by attachment_simple in the field unchanged[0]["filename"] I have the original file path, is that normal? Should it not be the zotero storage path /path/to/Zotero/storage?
My output looks like this:
{
  "success": [],
  "failure": [],
  "unchanged": [
    {
      "itemType": "attachment",
      "linkMode": "imported_file",
      "title": "mypdf.pdf",
      "accessDate": "",
      "note": "",
      "tags": [],
      "collections": [],
      "relations": {},
      "contentType": "",
      "charset": "",
      "filename": "/path/to/mypdf.pdf",
      "md5": "None",
      "mtime": "None",
      "parentItem": "WKC6M5SL",
      "key": "XR23RHQI"
    }
  ]
}

I have the original file path, is that normal? Should it not be the zotero storage path /path/to/Zotero/storage

No, the file path that's returned should match your local file path. But from your output above, the dict is in the wrong key: it's in unchanged, and if the file is successfully uploaded (and possibly attached to a parent item), it should be under success.


[CI] musl build updates

Separate job for musl with alpine:latest container
Cleanup of Makefiles and shell scripts

Signed-off-by: Azeem Sajid<EMAIL_ADDRESS>

@iamazeem could you also double-check if these contemplated changes have any measurable impact on the performance benchmarks?

@liquidaty: Here are the results:

Benchmarks (WITHOUT_SIMD=true)

Timestamp UTC: 2024-11-02T07:23:57Z

Releases Used

https://github.com/liquidaty/zsv/releases/download/v0.3.9-alpha/zsv-0.3.9-alpha-amd64-linux-musl.tar.gz
https://github.com/eBay/tsv-utils/releases/download/v2.2.0/tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz
https://github.com/BurntSushi/xsv/releases/download/0.13.0/xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz

Results
count
1 | zsv : real 0.17 user 0.16 sys 0.00
2 | zsv : real 0.17 user 0.17 sys 0.00
3 | zsv : real 0.17 user 0.17 sys 0.00
4 | zsv : real 0.17 user 0.16 sys 0.01
5 | zsv : real 0.18 user 0.17 sys 0.00
6 | zsv : real 0.17 user 0.17 sys 0.00
1 | xsv : real 0.08 user 0.07 sys 0.01
2 | xsv : real 0.08 user 0.08 sys 0.00
3 | xsv : real 0.08 user 0.07 sys 0.01
4 | xsv : real 0.08 user 0.08 sys 0.00
5 | xsv : real 0.08 user 0.08 sys 0.00
6 | xsv : real 0.08 user 0.07 sys 0.01
1 | tsv : real 0.08 user 0.08 sys 0.00
2 | tsv : real 0.08 user 0.09 sys 0.00
3 | tsv : real 0.09 user 0.06 sys 0.02
4 | tsv : real 0.08 user 0.08 sys 0.00
5 | tsv : real 0.08 user 0.07 sys 0.01
6 | tsv : real 0.09 user 0.08 sys 0.00

select
1 | zsv : real 0.39 user 0.36 sys 0.02
2 | zsv : real 0.42 user 0.41 sys 0.00
3 | zsv : real 0.39 user 0.39 sys 0.00
4 | zsv : real 0.39 user 0.37 sys 0.01
5 | zsv : real 0.39 user 0.39 sys 0.00
6 | zsv : real 0.38 user 0.38 sys 0.00
1 | xsv : real 0.36 user 0.36 sys 0.00
2 | xsv : real 0.36 user 0.35 sys 0.01
3 | xsv : real 0.36 user 0.35 sys 0.00
4 | xsv : real 0.36 user 0.35 sys 0.00
5 | xsv : real 0.36 user 0.36 sys 0.00
6 | xsv : real 0.36 user 0.34 sys 0.01
1 | tsv : real 0.11 user 0.10 sys 0.00
2 | tsv : real 0.11 user 0.10 sys 0.00
3 | tsv : real 0.11 user 0.10 sys 0.00
4 | tsv : real 0.11 user 0.10 sys 0.00
5 | tsv : real 0.11 user 0.10 sys 0.00
6 | tsv : real 0.11 user 0.08 sys 0.02

Benchmarks (WITHOUT_SIMD=false)

Timestamp UTC: 2024-11-02T07:52:20Z

Releases Used

https://github.com/liquidaty/zsv/releases/download/v0.3.9-alpha/zsv-0.3.9-alpha-amd64-linux-musl.tar.gz
https://github.com/eBay/tsv-utils/releases/download/v2.2.0/tsv-utils-v2.2.0_linux-x86_64_ldc2.tar.gz
https://github.com/BurntSushi/xsv/releases/download/0.13.0/xsv-0.13.0-x86_64-unknown-linux-musl.tar.gz

Results
count
1 | zsv : real 0.03 user 0.03 sys 0.00
2 | zsv : real 0.03 user 0.03 sys 0.00
3 | zsv : real 0.03 user 0.03 sys 0.00
4 | zsv : real 0.03 user 0.03 sys 0.00
5 | zsv : real 0.03 user 0.03 sys 0.00
6 | zsv : real 0.03 user 0.03 sys 0.00
1 | xsv : real 0.08 user 0.07 sys 0.00
2 | xsv : real 0.08 user 0.07 sys 0.01
3 | xsv : real 0.08 user 0.07 sys 0.00
4 | xsv : real 0.08 user 0.07 sys 0.00
5 | xsv : real 0.08 user 0.07 sys 0.01
6 | xsv : real 0.08 user 0.07 sys 0.00
1 | tsv : real 0.08 user 0.07 sys 0.00
2 | tsv : real 0.08 user 0.08 sys 0.00
3 | tsv : real 0.08 user 0.08 sys 0.00
4 | tsv : real 0.08 user 0.08 sys 0.00
5 | tsv : real 0.08 user 0.08 sys 0.00
6 | tsv : real 0.08 user 0.08 sys 0.00

select
1 | zsv : real 0.25 user 0.23 sys 0.01
2 | zsv : real 0.25 user 0.25 sys 0.00
3 | zsv : real 0.25 user 0.24 sys 0.01
4 | zsv : real 0.25 user 0.25 sys 0.00
5 | zsv : real 0.25 user 0.25 sys 0.00
6 | zsv : real 0.25 user 0.24 sys 0.00
1 | xsv : real 0.36 user 0.35 sys 0.00
2 | xsv : real 0.37 user 0.35 sys 0.01
3 | xsv : real 0.37 user 0.36 sys 0.00
4 | xsv : real 0.36 user 0.34 sys 0.02
5 | xsv : real 0.36 user 0.34 sys 0.01
6 | xsv : real 0.37 user 0.36 sys 0.00
1 | tsv : real 0.11 user 0.10 sys 0.00
2 | tsv : real 0.11 user 0.10 sys 0.00
3 | tsv : real 0.11 user 0.10 sys 0.00
4 | tsv : real 0.11 user 0.11 sys 0.00
5 | tsv : real 0.11 user 0.10 sys 0.00
6 | tsv : real 0.11 user 0.10 sys 0.00

Notes

zsv musl build without SIMD is from this workflow run (https://github.com/liquidaty/zsv/actions/runs/11633809775).
zsv musl build with SIMD is from this workflow run (https://github.com/liquidaty/zsv/actions/runs/11641005858).
For tsv-utils, the latest Linux build isn't available for its latest release (v2.2.1) . Using its previous release (v2.2.0) instead.

A binary built with SIMD instructions fails with "illegal instruction" error on platforms where those instructions aren't available.
We don't necessarily have to publish the non-SIMD release.
The WITHOUT_SIMD flag is just there to handle such scenarios whenever needed.

@iamazeem I'm not aware of any chips from 2018+ that do not have some sort of SIMD capabilities, so is it possible that WITHOUT_SIMD is using unnecessarily conservative assumptions, and the better solution would be to better tailor the build to properly discover and use the available SIMD instructions? Alternatively is there a limitation to the musl compiler's ability to generate correct SIMD instructions for varied platforms?

A binary built with SIMD instructions fails with "illegal instruction" error on platforms where those instructions aren't available. We don't necessarily have to publish the non-SIMD release. The WITHOUT_SIMD flag is just there to handle such scenarios whenever needed.

Disregard my prior comment, as I did not closely read the above-- thank you, makes sense. Let's include the WITHOUT_SIMD for CI, but not actually publish it (at least for now) upon release. Thank you!

@iamazeem lmk if this is ready to go and I will merge

Benchmarks from the latest run on main after merge:
https://github.com/liquidaty/zsv/actions/runs/11697708382


New protocol # 456

🥼 Protocol Request
Great, we just need the ID number and its title; fill in below:
Protocol ID:  456
Protocol Title: GLNE 007

Added protocol GLNE 007, James GLNE 007 🍸


app_version not set to git describe when building with docker image espressif/idf:v4.4.1 inside a github actions workflow (IDFGH-7504)

Environment

Development Kit:      none
Module or chip used:  ESP32
IDF version (run git describe --tags to find it): v4.4.1
Build System:         idf.py
Operating System:     Linux
Using an IDE?: No
Power Supply:         Battery

Problem Description
I have a github actions workflow to build our esp-idf project similar to this:
name: CI

on:
  push:
    branches:
      - master
  pull_request:
    branches:
      - master

  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
        with:
          submodules: 'recursive'

      - name: Build with esp-idf v4.4.1
        shell: bash
        run: docker run --rm -v $(pwd):/project -w /project espressif/idf:v4.4.1 idf.py build

      - name: Upload Build Artifacts
        uses: actions/upload-artifact@v2
        with:
          name: build
          path: |
            ${{ env.ESP_APP_FOLDER }}/build/
            !**/esp-idf/

Upon updating from espressif/idf:v4.3.1 to espressif/idf:v4.4.1, the app_version is no longer set to the output of git describe, it falls back to "1".
It may have something to do with git describe failing inside the container as attempting to use an interactive version of the image (docker run --rm -v $(pwd):/project -w /project/application -it espressif/idf:v4.4.1) on a Ubuntu WSL2 instance results in:
fatal: unsafe repository ('/project' is owned by someone else)
To add an exception for this directory, call:

        git config --global --add safe.directory /project

I attempted some of the workarounds discussed here with no results.
Expected Behavior
app_version is set to the output of git describe --always --tags --dirty
Actual Behavior
app_version is set to 1
Steps to reproduce

Set up git workflow similar to one mentioned above
Run git workflow to build esp idf app binary and upload to workflow artifacts.
Check resulting artifact app binary for app_version.

Debug Logs
This is a snippet of the logs from the github actions step:
Done! You can now compile ESP-IDF projects.
Go to the project directory and run:

  idf.py build

-- Found Git: /usr/bin/git (found version "2.25.1")
-- ccache will be used for faster recompilation
-- The C compiler identification is GNU 8.4.0
-- The CXX compiler identification is GNU 8.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /opt/esp/tools/xtensa-esp32-elf/esp-20[21](https://github.com/Oto-Lawn-Care/OtO_FW_3.0/runs/6679544975?check_suite_focus=true#step:5:22)r2-patch3-8.4.0/xtensa-esp[32](https://github.com/Oto-Lawn-Care/OtO_FW_3.0/runs/6679544975?check_suite_focus=true#step:5:33)-elf/bin/xtensa-esp32-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /opt/esp/tools/xtensa-esp32-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /opt/esp/tools/xtensa-esp32-elf/esp-2021r2-patch3-8.4.0/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- Building ESP-IDF components for target esp32
-- Project sdkconfig file /project/application/sdkconfig
-- Found Python3: /opt/esp/python_env/idf4.4_py3.8_env/bin/python3.8 (found version "3.8.10") found components: Interpreter
-- Performing Test C_COMPILER_SUPPORTS_WFORMAT_SIGNEDNESS
-- Performing Test C_COMPILER_SUPPORTS_WFORMAT_SIGNEDNESS - Success
-- App "project" version: 1

Note lines:
-- Project is not inside a git repository, or git repository has no commits; will not use 'git describe' to determine PROJECT_VER.
-- App "project" version: 1
Other items if possible
N/A

I've found a temporary workaround in the git actions workflow:
      - name: Add safe directory to global git config
        shell: bash
        run: git config --global --add safe.directory /project

      - name: Build with esp-idf v4.4.1
        shell: bash
        run: |
          docker run -v ~/.gitconfig:/etc/gitconfig -v $(pwd):/project -w /project/${{ env.ESP_APP_FOLDER }} espressif/idf:v4.4.1 idf.py build

The first step adds /project as a safe directory to the global git config, which is stored in ~/.gitconfig.
During the docker run, the global git config is mounted to the idf container which allows it to see the safe.directory setting and allow it to run git describe.

You could use https://github.com/espressif/esp-idf-ci-action which allows specifying the version, it uses a similar approach of mapping directories into the docker. I can confirm that the version is correctly picked up from git using the esp-idf-ci-action action.

@TonyWu98  can you share the PyOto repo? It is missing from https://github.com/Oto-Lawn-Care/


class java.lang.Process is not a value

I am running into an error with the following minimal code:
import cats.effect._

object Main extends IOApp {
  override def run(args: List[String]): IO[ExitCode] = {
    Process[IO]("ls", List("-hal"))
    IO(ExitCode.Success)
  }
}

I am getting:
❯ sbt run
[info] Loading global plugins from /Users/ethan/.sbt/1.0/plugins
[info] Loading settings for project stackoverflow-build from plugins.sbt ...
[info] Loading project definition from /Users/ethan/IdeaProjects/stackoverflow/project
[info] Loading settings for project stackoverflow from build.sbt ...
[info] Set current project to stackoverflow (in build file:/Users/ethan/IdeaProjects/stackoverflow/)
[info] Compiling 1 Scala source to /Users/ethan/IdeaProjects/stackoverflow/target/scala-2.13/classes ...
[error] /Users/ethan/IdeaProjects/stackoverflow/src/main/scala/Main.scala:5:5: class java.lang.Process is not a value
[error]     Process[IO]("ls", List("-hal"))
[error]     ^
[error] one error found
[error] (Compile / compileIncremental) Compilation failed
[error] Total time: 2 s, completed Apr 30, 2020, 6:29:18 PM

This is my build.sbt:
fork := true

name := "stackoverflow"

version := "0.1"

scalaVersion := "2.13.2"

libraryDependencies += "io.github.vigoo" %% "prox" % "0.5.1"

What am I missing?

The import statement! lol:
import io.github.vigoo.prox._
import io.github.vigoo.prox.syntax._


fix: allows retries when setting maxAttempts without setting TotalTimeout

Fixes #827

Codecov Report

Merging #835 into master will increase coverage by 0.03%.
The diff coverage is 100%.

@@             Coverage Diff              @@
##             master     #835      +/-   ##
============================================
+ Coverage     78.72%   78.75%   +0.03%
- Complexity     1146     1148       +2
============================================
  Files           202      202
  Lines          5099     5103       +4
  Branches        405      404       -1
============================================
+ Hits           4014     4019       +5
  Misses          912      912
+ Partials        173      172       -1

Impacted Files
Coverage Δ
Complexity Δ

...le/api/gax/retrying/ExponentialRetryAlgorithm.java
100% <100%> (+2.43%)
12 <0> (+2)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 68ae22d...b4b2236. Read the comment docs.

Looking at the original issue it is not clear why the decision was made to modify the algorithm itself but not how the retry settings builder behaves.

I think the underlying issue is that the algorithm assumes that there is always a total timeout, which is why it  needs to be updated

@igorbernstein2 @rahulKQL I think if the case when both maxAttempts == 0 & totalTimeout == 0  would mean that the retries are disabled then I would be Ok with this PR. Also please update documentation for retry settings to explain that situation.

@rahulKQL can you update the RetrySettings docs? I think updating this this paragraph from
 *
The "total timeout" parameter has ultimate control over how long the logic should keep trying
 * the remote call until it gives up completely. The higher the total timeout, the more retries can
 * be attempted. The other settings are considered more advanced.

To:
 *
The "total timeout" and "max attempts" settings have ultimate control over how long the logic should keep trying the remote call until it gives up completely. The remote call will be retried until one of those thresholds is crossed. To avoid unbounded rpc calls, it is required to configure one of those settings. If both are 0, then retries will be disabled. The other settings are considered more advanced.

@rahulKQL can you also update the definition of
https://github.com/googleapis/gax-java/blob/8d7aafcd012d83b85f8733b6ef589fcbbf2ee261/gax/src/main/java/com/google/api/gax/rpc/Callables.java#L249-L251?

@igorbernstein2 @vam-google Thanks for the review, I have updated this change request accordingly. Could you please have a fresh look!


Skip yaml validation for files that are completely unchanged

Thanks to https://github.com/dbt-labs/dbt/pull/3460, when dbt reads yaml files, it also performs minimal validation and stores their intermediate representation in dictionary form. This enables us to:

Return good error messages early & often
Better separate file loading from manifest construction
In partial parsing: only re-parse nodes affected by specific lines changed in yaml files, rather than re-parsing all resources implicated in the entire yaml file

At the same time, partial parsing runs need to read and validate every yaml file in the project, every time. This should still be really fast, even for big projects, so long as the user has libyaml installed (C bindings for PyYAML). I'm going to make sure we document this.
Would it be possible to have the best of both worlds, by storing a hash of the yaml file, and skip re-reading / validating it entirely if it's completely unchanged?

@gshank would love to talk more about this when you're back next week!

We do store a yaml dictionary now (and the checksum), but those are in the partial parse msgpack file, and we have no access to it when the files are read. And it's only available when partial parsing is turned on. We could move that validation to the partial parsing code, but that does mean that the error messages are different from the messages issued by the schema parser. We could add that same minimal validation to the schema parser too, so that the errors are different. I do think that eventually we're going to want to have validation that's not derived from hologram, since those error messages are sometimes cryptic and hard to decipher. So we could consider that a first step in that direction.

@gshank Thanks for weighing in! I was curious if there was a quick win here that could prevent the need to re-read + re-validate thousands of yaml files if zero files have been edited. It sounds like this would be a slightly bigger lift.
At the same time, we also believe that, so long as the user/deployment environment has optimized libraries installed, re-reading + re-validating thousands of yaml files should be very, very fast. Does that sound right?
(cc @drewbanin)

I spent some time today diving into the raw load-time data a bit more, and I'm seeing that in parsing steps where partial parsing "works" (parsed_path_count < path_count), the read_files step comprises a huge portion of the total load time, between 80-90%. File reading by itself can take 2-7 ms per file. In very large projects, this feels like the big thing standing between our pretty-darn-good v0.20 numbers (3-9 ms/path with partial parsing), versus our end-of-2021 goal of 1 ms / file, <5 s total in a 5k file project.
It feels like some reasonable next steps could be:

Add tracking for whether users have the optimized libyaml installed, to validate whether this is a crucial differentiator. Ensure that dbt Cloud is using the optimized C yaml bindings (cc @leahwicz)
Following the proposal in this issue, store file hashes in partial_parse.msgpack and use that as a shortcut for file reading, even if it means some light refactoring and separation/duplication of validation steps

big +1 here - totally non-scientific but I also dug into the partial parsing perf data and found this same thing - dbt is doing a lot of (slow) work to determine that two .yml files are identical :)

@kwigley opened https://github.com/dbt-labs/dbt-cloud/pull/4018 to get the optimizedlibyaml into Cloud
I opened this issue to get the tracking work done as well: https://github.com/dbt-labs/dbt/issues/3845

In order to skip loading the files we'd need to look at the file timestamp instead of the checksum, because we can't get the checksums without reading the files. Is this something that we're comfortable doing?


MX250 don't need to install OpenCL SDK as being integrated presumably in the driver

I installed CUDA in the first place, and it works. Then I read another tutorial where they use oclDeviceQuery to check for openCl info, by which I found Opencl 1.2 and CUDA 10.1 are right there even after I uninstalled CUDA 10.0. The python script still runs without problem.

Hi, D2Joy. Thanks for your information. Yes, you discovered a relatively confusing situation. In fact, the OpenCL code can run without any OpenCL SDK because running them only needs OpenCL runtime, which has been always integrated in normal graphics driver for these recently-released GPUs like MX250. However, you need OpenCL SDK to build the pyOpenCL for your GPU platform. Once the build complete, you don't need OpenCL SDK anymore.

Oh, Right. The OpenCL method is undoubtedly superior than CPU one. Am I right to see it as the first option for general password bruteforce and do you mind my citing it elsewhere?

Of course you can cite this repo. The OpenCL is indeed an important tool for GPU programming, but the CUDA language is also important. This repo cannot brute force other kind of encryption. Up to now, it can only deal with AES-256-CBC encryption with PBKDF2 and no HMAC.  To get a large variety of password bruteforcing code in OpenCL, you can go to the Hashcat website.


@babel/preset-env/data/built-ins.json has been removed from babel

Version
v2.4.5
Reproduction link
https://qx9nql3y49.sse.codesandbox.io/
Steps to reproduce
Just follow latest nuxtjs installation procedure.
What is expected ?
nuxtjs running
What is actually happening?
client build failed

This bug report is available on Nuxt community (#c8869)

Exactly same problem. Cant create
npx create-nuxt-app

I'm having the same problem here.

I'm having this problem as well, both on new projects using npx create-nuxt-app, but also on fresh installs of my existing nuxt app.

It seems that a fix has been merge https://github.com/babel/babel/pull/9711

npm install --save-dev @babel/preset-env
fixed my latest project

Here I was thinking it was just me.  I chose a hell of a time to decide to try Nuxt heh.
@scottyadean this was my first instinct, but it didn't solve the problem.

@vsergiu93 after applying the fix, I'm still receiving isPluginRequired is not a function.  I managed to get a function by importing directly from @babel/preset-env/lib/filter-items but this yielded an error of no property "ie".
A little lost.

@kristremblay All I did was to make sure I have updated @babel/preset-env package to 7.4.1 then I ran yarn upgrade

@vsergiu93 Just deleted node_modules, updated package.json, and installed from scratch.  Works now, thanks :)

Thanks folks for reporting this issue and closing this as there seems a bad release from babel caused that and there is workaround for this issue.

@pi0 It might be helpful to rollback https://github.com/nuxt/nuxt.js/commit/f5f1471bc5d205064e4c10e9c956ba6e88721975 until this gets addressed appropriately. I imagine that it will be resolved in the following days so requiring developers to add the workaround seems more detrimental than helpful.

@dasZGFz downgrading in package.json does not simply fixes this issue unless using ~7.3.4 constraint.

@pi0 That commit also alters the yarn.lock file. As far as I'm aware, that's the breaking commit so I would assume reverting it would fix the issue.
It's entirely possible that I'm overlooking something but I don't think I am.

The yarn.lock is NOT published with npm packages but only used for nuxt development and CI tests. What babel version users get installed with fresh create-nuxt-app is the latest version of babel that this constraint allows which is 7.4.0.
Please follow-up other answers by cleaning up your project's lock file and installing again.

The "With useBuiltIns option, required direct setting of corejs option" warning is going to be resolved by #5291. (Thanks to @clarkdo)

@pi0 Ah I see. Thank you for clarifying. What I was looking for was locking the babel version, not the yarn.lock file itself.
Glad to see a PR already emerge aiming to resolve this issue. Thanks for the hard work!

Resolved by #5291 :relaxed:

For the users who are using yarn and need a quick fix before next Nuxt release:
Add resolutions in package.json
"resolutions": {
  "@babel/preset-env": "7.3.1"
}

Babel 7.4.0 support is resolved with v2.5.1. You can now safely remove the resolutions field.


Issues with media playback in Firefox

Hello, it seems that recently any media (audio&video) playback in Firefox became broken with exploit guard settings provided in this repository.
I checked that any of these options trigger this behavior:
EnableExportAddressFilter
EnableExportAddressFilterPlus
EnableImportAddressFilter
EnableRopStackPivot
EnableRopCallerCheck
EnableRopSimExec

So working settings are
Set-ProcessMitigation -Name firefox.exe -Enable DEP,BottomUp,ForceRelocateImages,CFG,AuditRemoteImageLoads,AuditLowLabelImageLoads,SEHOP,TerminateOnError,AuditChildProcess
Checked in Firefox 93.0 x64 and Windows 10 19043.1237

It seems that almost all of these are working again in Firefox 94.0.2:
EnableExportAddressFilter EnableExportAddressFilterPlus EnableImportAddressFilter EnableRopStackPivot EnableRopCallerCheck EnableRopSimExec
But "EnableExportAddressFilterPlus" rule is now crashing Firefox all the time just on startup.
Working settings are:
Set-ProcessMitigation -Name firefox.exe -Enable DEP,BottomUp,ForceRelocateImages,CFG,AuditRemoteImageLoads,AuditLowLabelImageLoads,EnableExportAddressFilter,EnableImportAddressFilter,EnableRopStackPivot,EnableRopCallerCheck,EnableRopSimExec,SEHOP,TerminateOnError,AuditChildProcess
Firefox 94.0.2 x64, Windows 10 19043.1348


Deprecate posix timer

Brief feature description
Deprecate posix timer from iceoryx code base.
Detailed information
Posix timer has several issues (concurrency, truncation) and is pretty heavy weight and a bit slow. As iceorxy needs to be very responsive the posix timer should be removed and replaced with a lightweight stopwatch.
This will also close #190 as it'll be obsolete.
Tasks

[x] create DeadlineTimer or similar and replace this part of the PosixTimer where applicable
[x] create an object that creates a thread and peridically executes a task
[ ] create a small wrapper around a posix::semaphore which can be used for concurrent::PeriodicTask
[ ] replace the keep alive message from a std::chrono timestamp with a units::Duration from a timespec initialized with clock_gettime with CLOCK_REALTIME -> maybe better to use a different issue for this

@marthtz I think a stopwatch is not exactly what we need. We have to rewrite the timer so that the timer does not work with callbacks anymore but instead with signals. If you look at the manpage https://man7.org/linux/man-pages/man2/timer_create.2.html we have to possible approaches:

send a signal to the whole process
send a signal to a specific thread specified when creating the timer

I think we should explore the second option. Maybe it has similar drawbacks than the callback approach but if not it should be our way to go.
@elBoberido what do you think?

@elfenpiff @marthtz please don't call it stopwatch. A stopwatch is measuring time, this functionality is running towards a deadline or countdown. See also the equivalent int boost and Qt.
To the other points. The posix timer was started as a deadline timer and then enhanced with the posix calls. Currently it has both functionalities.

cut out the deadline functionality and use that class where appropriate
deprecate the current posix timer and replace the remaining occurrences with std::thread and sleep_for
decide if we really need something like the posix timer. If we do, implement the second option from @elfenpiff which is also described in more detail in #170

@elfenpiff I am currently working on this story. Based on the comments from @dkroenke I am working on to replace the timer.start() function with a simple thread sleep. Currently the posixtimer.start() is being called within the posh_runtime, service_discovery_notifier.

@shankar-bosch a simple sleep_for is not enough. The current timer invokes a thread on each timeout. You also have a start a thread where you use the sleep_for.

@elBoberido okay I will do so

@shankar-bosch if you are working on an issue could you please assign it to you and add it into the iceoryx 1.0 board: https://github.com/eclipse/iceoryx/projects/2 so that everyone is aware of it.

@elfenpiff sure. I am not in the list of contributors. Could you pls add me? I can assign this issue in my name and bring this to the board.

@elBoberido I have clarifications. As you know the timers can be one shot or periodic based on the it_interval .
If it is a one shot timer there can be a thread::sleepfor(it_value).
if the timer is a periodic then there has to be a thread::sleepfor(it_value) followed by a thread::sleepfor(it_interval) until the application exit? I am not sure what to do here.
I also dont see the start() function within the timer.cpp invoking a thread after a timer is disarmed. Is it done in a different place? could you pls also help me here?

@shankar-bosch if you are working on an issue could you please assign it to you and add it into the iceoryx 1.0 board: https://github.com/eclipse/iceoryx/projects/2 so that everyone is aware of it.

@elfenpiff This can only be done by an Eclipse committer. Eclipse committers are voted on in elections.

@mossmaurice Thank you

@shankar-bosch a simple sleep_for is not enough. The current timer invokes a thread on each timeout. You also have a start a thread where you use the sleep_for.

How about just removing the timer from iceoryx ?
I would need to double check but the current use-cases we have within iceoryx can be resolved using a simple sleep_for (e.g. sending heartbeat).

@ithier that's the idea of this issue. There is an additional functionality in the timer (checking for a deadline) and that should to be preserved

@shankar-bosch the thread is started implicitly. Have a look at https://linux.die.net/man/2/timer_create, we use SIGEV_THREAD
If you use just sleep_for, the semantic of the code changes. In this case the thread which previously called start is now blocked, while it just continues running with the current timer implementation. You have to create a thread and use sleep_for in a loop. I would suggest to start with cutting out the functionality of the DeadlineTimer to cxx/deadline_timer.hpp/cpp. This should be way easier. You can do it by copying the current timer to cxx and removing everything from it which is related to the m_osTimer member

Here is where the timer is used in iceoryx:

sending heartbeat signal -> can be replaced by creating a single extra thread that calls sleep_for
waiting for roudi up to a deadline -> can be replaced by saving epoch time and just comparing to the current time each iteration, this does not need another thread

From above, it looks like we only need some kind of helper method to wrap the usage of timer_gettime to satsify the second usecase.
Then we can just mark the old timer as deprecated and we are done with this issue, right ?

it just occurred to me that sleep_for might be really bad for the test time, since it's not interruptible. How about creating a small wrapper around a semaphore with timed wait?
e.g.
class Timer {
public:
    Timer() noexcept = default;
    ~Timer() noexcept; // calls stop
    cxx::expected<TimerError> wait(cxx::Duration) noexcept; // blocking wait on the semaphore
    void stop() noexcept; // sem_post to interrupt waiting
    static cxx::expected<units::Duration, TimerError> Timer::now() noexcept; // this is from the old Timer and also used in the codebase
}

Later on the semaphore could be replaced by POSIX timer_create(..) etc, if necessary.
What do you think?

Current PR #333 also makes use of the timer. Basically, it's checking for a timeout / interval expiry.
@elBoberido wrapper sounds good to me and could be used for this and @ithier first bullet point.

@elBoberido Thanks for the wrapper solution.

We should definitely fix this for 1.0, On ARM 64 I have failing tests because of #190 but I fear that's not the only problem

@budrus to fix #190 we need to refactor the units::Duration away from long double to chrono or timespec/timeval

@elBoberido That's at least half of the game. The failing DestructorIsBlocking could maybe be another thing. This also fails with larger numbers...

@budrus it could still be a rounding issue. AFAIK, the timer doesn't sleep for e.g. 100ms but for currentTime + 100ms. This might result in rounding errors with the long double, especially since that's only a alias to double on ARM and 80 bit wide on x86

@budrus to fix #190 we need to refactor the units::Duration away from long double to chrono or timespec/timeval

@elBoberido @budrus I suggest here to use neither of them. I would create a struct which looks like timespec
struct duration_t {
     uint64_t seconds;
     uint32_t nanoseconds;
};

The reason is that a duration cannot be negative, you cannot wait -5 seconds but timespec/timeval can be negative. Additionally, timespec/timeval do not have a fixed size they are using the types long and time_t which can differ on different architectures.
And when we refactor units::Duration we should get rid of all the long double operators which are absolutely unnecessary - and I know I was the one who introduced them in the first place :/ .

I will take the solution for the truncation issue as suggested by @elfenpiff. Thank you. @budrus Is there a seperate ticket for failing DestructorIsBlocking issue?

@shankar-bosch No, as @elBoberido said,  it could also have the same root cause as #190. We would have to analyze this but now things are changing anyway

@budrus Okay. I believe the issue #190 fix should be available for 1.0. So increasing the scope of #337 to fix #190 and analysing the failing DestructorIsBlocking should be fine?

since we are approaching the 0.90 release, is there any update on this?

Yes the PR is getting ready. Should be raised shortly.

@shankar-bosch for Deadline or keep-alive?

Deadline is ready. Keep alive is under testing. The PR should cover both.

@shankar-bosch do you think you can get this in until monday? If not, I could do a fast fix for the keep alive.
Just as reminder to keep the PRs small. The PR you are preparing could be split into two. The dealine timer and the keep alive functionality.

@elBoberido I have my changes ready. But I havent got the committer approval to open the PR yet. Yes you are right I can split the PR into two. shall I raise the PR for the deadline today by forking other repo? I shall also check this with my PO and update you.

@shankar-bosch yes, please. Would you mind if I create a small workaround for the keep-alive?

@shankar-bosch I've created a PeriodicTask class. Currently it wraps a semaphore. This semaphore could be replaced by a refactored posix::Timer. Ideally the timer doesn't spawn a thread, but does just block on wait. This would make the timer more flexible since it could also be used when we want to block on the current thread.

@elBoberido Thank you for the changes. I shall adopt the change.

@shankar-bosch with you DeadlineTimer PR slowly merging, there is only one remaining part to refactor.
The old posix::Timer had three features:

DeadlineTimer -> refactored out
spawning a thread and execute a task -> moved to PeriodicTask
periodically trigger an event -> currently also in PeriodicTask with the Semaphore

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

I think the current RunMode can be omitted, since you can do it by calling wait() only once or repeatedly. I'm not sure about the CatchUpPolicy but I think it should also be a parameter of the constructors or maybe even better of the wait() method.
@mossmaurice @MatthiasKillat what do you think?
If you have time and like the task, feel free to implement it. Ideally, one PR for the timer refactoring and one for the integration into PeriodicTask.

@shankar-bosch with you DeadlineTimer PR slowly merging, there is only one remaining part to refactor.
The old posix::Timer had three features:

DeadlineTimer -> refactored out
spawning a thread and execute a task -> moved to PeriodicTask
periodically trigger an event -> currently also in PeriodicTask with the Semaphore

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

I think the current RunMode can be omitted, since you can do it by calling wait() only once or repeatedly. I'm not sure about the CatchUpPolicy but I think it should also be a parameter of the constructors or maybe even better of the wait() method.
@mossmaurice @MatthiasKillat what do you think?
If you have time and like the task, feel free to implement it. Ideally, one PR for the timer refactoring and one for the integration into PeriodicTask.

@shankar-bosch with you DeadlineTimer PR slowly merging, there is only one remaining part to refactor.
The old posix::Timer had three features:
* DeadlineTimer -> refactored out

* spawning a thread and execute a task -> moved to PeriodicTask

* periodically trigger an event -> currently also in PeriodicTask with the Semaphore

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

I think the current RunMode can be omitted, since you can do it by calling wait() only once or repeatedly. I'm not sure about the CatchUpPolicy but I think it should also be a parameter of the constructors or maybe even better of the wait() method.
@mossmaurice @MatthiasKillat what do you think?
If you have time and like the task, feel free to implement it. Ideally, one PR for the timer refactoring and one for the integration into PeriodicTask.

@elBoberido Sure. I am interested to take up. I have some points for clarifications. Shall i create a new story for this refactoring?

@shankar-bosch with you DeadlineTimer PR slowly merging, there is only one remaining part to refactor.
The old posix::Timer had three features:
* DeadlineTimer -> refactored out

* spawning a thread and execute a task -> moved to PeriodicTask

* periodically trigger an event -> currently also in PeriodicTask with the Semaphore

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

I think the current RunMode can be omitted, since you can do it by calling wait() only once or repeatedly. I'm not sure about the CatchUpPolicy but I think it should also be a parameter of the constructors or maybe even better of the wait() method.
@mossmaurice @MatthiasKillat what do you think?
If you have time and like the task, feel free to implement it. Ideally, one PR for the timer refactoring and one for the integration into PeriodicTask.

@elBoberido Sure. I am interested to take up. I have some points for clarifications. Shall i create a new story for this refactoring?

@shankar-in from my point of view, this issue could just be renamed to Refactor posix timer. In the end, that's what we did all the time

@shankar-in from my point of view, this issue could just be renamed to Refactor posix timer. In the end, that's what we did all the time

Thanks @shankar-in and @elBoberido I think we already made good progress.
Throughout this post I'm referring not to master but the current branch of #490

* spawning a thread and execute a task -> moved to PeriodicTask

IMHO that's a great building block. I would keep the semaphore out of there. Could be renamed to PeriodicThread as it is just syntactic sugar as you already noted. The name "task" is probably the most über-undefined term every in automotive software development. And probably elsewhere, too. Avoiding it could be wise.
Use-case: Do something ASAP periodically, no need for the timing to be precise e.g. MQ/UDS handling of keep-alive messages.

* periodically trigger an event -> currently also in PeriodicTask with the Semaphore

For me that's a separate entity. Let's call it PeriodicEvent.

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

Let's recap a bit...
Lessons learned:

posix::timer was waaaay to complex and contained different kind of constructs in one

Letting the OS start a thread was a bad idea, as in some occasions thousands of threads were spawned

restart() is unnecessary

I like your proposal, it keeps things simple. However, I'm unsure on whether a plain wait() will be user friendly enough. The idea of the legacy posix::Timer was to make it user-friendly by storing the callable inside the posix::Timer. #473 might come in handy here.
Around mid-2020 as a possible alternative to get rid of the too-many-thread-spawning-OS issue was discussed:

Stick with the API

struct sigevent has a member sigev_notify and could be set to SIGEV_SIGNAL
Don't use SIGEV_THREAD
OS then creates the "Event" an signals the process

Benefit: Timing is controlled by OS

I fear we might need at kernel-space solution at some point and a user-space solution won't be feasible e.g. to fulfill certain timing constraints on QNX. IIRC QNX advised to use the timer_create API. I also have the feeling that all CatchupPolicys should be supported by the new constructs.

@mossmaurice @MatthiasKillat what do you think?

Answering with a question: What would be the use-case for the PeriodicEvent?

Thanks @shankar-in and @elBoberido I think we already made good progress.
Throughout this post I'm referring not to master but the current branch of #490

* spawning a thread and execute a task -> moved to PeriodicTask

IMHO that's a great building block. I would keep the semaphore out of there. Could be renamed to PeriodicThread as it is just syntactic sugar as you already noted. The name "task" is probably the most über-undefined term every in automotive software development. And probably elsewhere, too. Avoiding it could be wise.
Use-case: Do something ASAP periodically, no need for the timing to be precise e.g. MQ/UDS handling of keep-alive messages.

* periodically trigger an event -> currently also in PeriodicTask with the Semaphore

For me that's a separate entity. Let's call it PeriodicEvent.

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

Let's recap a bit...
Lessons learned:

posix::timer was waaaay to complex and contained different kind of constructs in one

Letting the OS start a thread was a bad idea, as in some occasions thousands of threads were spawned

restart() is unnecessary

I like your proposal, it keeps things simple. However, I'm unsure on whether a plain wait() will be user friendly enough. The idea of the legacy posix::Timer was to make it user-friendly by storing the callable inside the posix::Timer. #473 might come in handy here.
Around mid-2020 as a possible alternative to get rid of the too-many-thread-spawning-OS issue was discussed:

Stick with the API

struct sigevent has a member sigev_notify and could be set to SIGEV_SIGNAL
Don't use SIGEV_THREAD
OS then creates the "Event" an signals the process

Benefit: Timing is controlled by OS

I fear we might need at kernel-space solution at some point and a user-space solution won't be feasible e.g. to fulfill certain timing constraints on QNX. IIRC QNX advised to use the timer_create API. I also have the feeling that all CatchupPolicys should be supported by the new constructs.

@mossmaurice @MatthiasKillat what do you think?

Answering with a question: What would be the use-case for the PeriodicEvent?

@mossmaurice

Thanks @shankar-in and @elBoberido I think we already made good progress.
Throughout this post I'm referring not to master but the current branch of #490

* spawning a thread and execute a task -> moved to PeriodicTask

IMHO that's a great building block. I would keep the semaphore out of there. Could be renamed to PeriodicThread as it is just syntactic sugar as you already noted. The name "task" is probably the most über-undefined term ever in automotive software development. And probably elsewhere, too. Avoiding it could be wise.
Use-case: Do something ASAP periodically, no need for the timing to be precise e.g. MQ/UDS handling of keep-alive messages.

I used task because it's also used in the STL https://en.cppreference.com/w/cpp/thread/packaged_task. I'm also not quite fond of PeriodicThread since that's not what it does. It's more like a PeriodicExecutor.

* periodically trigger an event -> currently also in PeriodicTask with the Semaphore

For me that's a separate entity. Let's call it PeriodicEvent.

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

Let's recap a bit...
Lessons learned:
* `posix::timer` was waaaay to complex and contained different kind of constructs in one

  * Letting the OS start a thread was a bad idea, as in some occasions thousands of threads were spawned

* `restart()` is unnecessary

I like your proposal, it keeps things simple. However, I'm unsure on whether a plain wait() will be user friendly enough. The idea of the legacy posix::Timer was to make it user-friendly by storing the callable inside the posix::Timer. #473 might come in handy here.

Well, it's just a building block. If you need the additional functionality use the PeriodicTask, which uses this building block. Fun fact, the PeriodicTask can store objects overloading the call operator ;)

Around mid-2020 as a possible alternative to get rid of the too-many-thread-spawning-OS issue was discussed:
* Stick with [the API](https://man7.org/linux/man-pages/man2/timer_create.2.html)

  * `struct sigevent` has a member `sigev_notify` and could be set to `SIGEV_SIGNAL`
  * Don't use `SIGEV_THREAD`
  * OS then creates the "Event" an signals the process

* Benefit: Timing is controlled by OS

I fear we might need at kernel-space solution at some point and a user-space solution won't be feasible e.g. to fulfill certain timing constraints on QNX. IIRC QNX advised to use the timer_create API. I also have the feeling that all CatchupPolicys should be supported by the new constructs.

The timer would currently use the semaphore, but that's just an implementation detail and can be changed in the future with POSIX timer_create functionality. I'm also for the CatchupPolicy and would pass it in the wait method. This has the benefit to define it when it's necessary, while waiting for the next timer tick.

@mossmaurice @MatthiasKillat what do you think?

Answering with a question: What would be the use-case for the PeriodicEvent?

For me an event is something active, and is fired no matter what. The proposed timer is completely passive since it just blocks on wait. Something like a PeriodicEvent could use the timer as building block though. That would also be the use case for the refactored timer, a building block for PeriodicTask. One could ask if that justifies a new class. I would say yes, since it nicely encapsulates the functionality and makes it available for cases where you already have a thread, need to block for a certain time and want to be able to interrupt the blocking call.

@mossmaurice

Thanks @shankar-in and @elBoberido I think we already made good progress.
Throughout this post I'm referring not to master but the current branch of #490

* spawning a thread and execute a task -> moved to PeriodicTask

IMHO that's a great building block. I would keep the semaphore out of there. Could be renamed to PeriodicThread as it is just syntactic sugar as you already noted. The name "task" is probably the most über-undefined term ever in automotive software development. And probably elsewhere, too. Avoiding it could be wise.
Use-case: Do something ASAP periodically, no need for the timing to be precise e.g. MQ/UDS handling of keep-alive messages.

I used task because it's also used in the STL https://en.cppreference.com/w/cpp/thread/packaged_task. I'm also not quite fond of PeriodicThread since that's not what it does. It's more like a PeriodicExecutor.

* periodically trigger an event -> currently also in PeriodicTask with the Semaphore

For me that's a separate entity. Let's call it PeriodicEvent.

I'd like to have the last one in a new posix::Timer, but based on a Semaphore, similar to the one in PeriodicTask but without spawning a new thread, just a blocking wait on e.g. a wait() method. This gives you the flexibility to use the timer in your current thread if you are allowed to block there, instead of spawning a new thread with PeriodicTask. I think of an API like this
class Timer {
    Timer(); // default c'tor
    Timer(units::Duration interval); // immediately starts the timer
    static cxx::expected<units::Duration, TimerError> now() noexcept;
    cxx::expected<TimerError> start(); // starts or restarts the timer with the current interval
    cxx::expected<TimerError> start(units::Duration interval); // starts or restarts the timer with the new interval
    cxx::expected<TimerError> stop(); // this is the only method that should be called from a different thread than the object was created
    cxx::expected<TimerEvent, TimerError> wait() noexcept; // blocking wait on the semaphore; with TimerEvent enum with the values `TICK` and `STOP`
};

Let's recap a bit...
Lessons learned:
* `posix::timer` was waaaay to complex and contained different kind of constructs in one

  * Letting the OS start a thread was a bad idea, as in some occasions thousands of threads were spawned

* `restart()` is unnecessary

I like your proposal, it keeps things simple. However, I'm unsure on whether a plain wait() will be user friendly enough. The idea of the legacy posix::Timer was to make it user-friendly by storing the callable inside the posix::Timer. #473 might come in handy here.

Well, it's just a building block. If you need the additional functionality use the PeriodicTask, which uses this building block. Fun fact, the PeriodicTask can store objects overloading the call operator ;)

Around mid-2020 as a possible alternative to get rid of the too-many-thread-spawning-OS issue was discussed:
* Stick with [the API](https://man7.org/linux/man-pages/man2/timer_create.2.html)

  * `struct sigevent` has a member `sigev_notify` and could be set to `SIGEV_SIGNAL`
  * Don't use `SIGEV_THREAD`
  * OS then creates the "Event" an signals the process

* Benefit: Timing is controlled by OS

I fear we might need at kernel-space solution at some point and a user-space solution won't be feasible e.g. to fulfill certain timing constraints on QNX. IIRC QNX advised to use the timer_create API. I also have the feeling that all CatchupPolicys should be supported by the new constructs.

The timer would currently use the semaphore, but that's just an implementation detail and can be changed in the future with POSIX timer_create functionality. I'm also for the CatchupPolicy and would pass it in the wait method. This has the benefit to define it when it's necessary, while waiting for the next timer tick.

@mossmaurice @MatthiasKillat what do you think?

Answering with a question: What would be the use-case for the PeriodicEvent?

For me an event is something active, and is fired no matter what. The proposed timer is completely passive since it just blocks on wait. Something like a PeriodicEvent could use the timer as building block though. That would also be the use case for the refactored timer, a building block for PeriodicTask. One could ask if that justifies a new class. I would say yes, since it nicely encapsulates the functionality and makes it available for cases where you already have a thread, need to block for a certain time and want to be able to interrupt the blocking call.

@elBoberido I am taking up refactoring of periodically trigger an event in this sprint. In case i have clarifications I will contact you in Gitter.

@elBoberido the abstimeout calculation for the semaphore is present within the semaphore.cpp. this makes the choice of clock unavailable for the the periodic timer. currently the semaphore always makes use of the system clock. For periodic timer I believe monotonic would be a right choice? In that case we have to move the calculation of the abstimeout outside the semaphore ? pls feedback

@shankar-in The implementation is absolutely correct here! We do not want to do this outside since it is error prone especially when the developer does not read the documentation of the posix semaphore. Here the part from the sem_timedwait manpage:
sem_timedwait() is the same as sem_wait(), except that abs_timeout specifies  a  limit  on
the  amount of time that the call should block if the decrement cannot be immediately per‐
formed.  The abs_timeout argument points to a structure that specifies an absolute timeout
in  seconds and nanoseconds since the Epoch, 1970-01-01 00:00:00 +0000 (UTC).  This struc‐
ture is defined as follows:
       struct timespec {
           time_t tv_sec;      /* Seconds */
           long   tv_nsec;     /* Nanoseconds [0 .. 999999999] */
       };

As you see we require here the unix time which is monotonic since it counts the seconds from 1970-01-01 (Timezone: UTC).

@elfenpiff  Thanks.
I do believe the time since Epoch depends on the current system time. The system time can be adjusted and if its adjusted the time since Epoch can deliver different results. However the monotonic clock is not affected by the system time. Isnt monotonic a better option? Pls feedback

@elfenpiff  Thanks.
CLOCK_REALTIME returns the Epoch based on the system time clock. So I think this can be modified which can lead to change in the time since epoch.
CLOCK_MONOTONIC doesnt depend on the system clock and cannot be set. So this value cannot be modified.
So isnt the Monotonic optimal for our periodic timer use case? pls feedback.

@shankar-in that depends. If you use the semaphore for the blocking wait then you have no choice and must use CLOCK_REALTIME. We cannot always decide which clock to use, sometimes the API we use takes this decision for us. So every time you use a POSIX API which takes a timespec or timeval you have to check which clock the API expects like elfenpiff already mentioned with https://linux.die.net/man/3/sem_timedwait

@elBoberido Okay. I am making use of the api sem_timedwait and let the API decide which would be the best clock for the purpose. :)

@elBoberido I have a clarification.
Pls find the impl of wait of the periodic / keep alive timer below,
`cxx::expected<iox::cxx::TimerEvent, posix::SemaphoreError> PeriodicTimer::wait() noexcept
{
if (*(m_stop.getValue()) == static_cast(posix::SemaphoreWaitState::TIMEOUT))
{
auto waitResult = m_stop.timedWait(m_interval, true);
    if (waitResult.has_error())
    {
        return cxx::error<posix::SemaphoreError>(waitResult.get_error());
    }
    else
    {
        return cxx::success<iox::cxx::TimerEvent>(iox::cxx::TimerEvent::TICK);
    }
}
else
{
    return cxx::success<iox::cxx::TimerEvent>(iox::cxx::TimerEvent::STOP);
}

}
The clarification I have is. The periodic timer will be manipulated by two different threads 1. Periodic Task 2. call back function thread. The periodic timerstart()/stop()functions will be called by the periodic task. The callback function thread will call thewait()on the periodic timer. Now Lets assume that Callback function thread is inwait()and blocked for the duration. Meanwhile if the periodic task threadstops()it releases the semaphore then thewait()` method acquires the semaphore immediately and keeps the timer Ticking.
So to avoid this concurrency issue should we introduce a mutex between the threads to update or read the semaphore state?
Pls feedback

@elBoberido Trying a work around. If the sem_ timedwait() didnt complete the wait duration then we can conveniently assume that semaphore is obtained [sem_post() happened in backgorund]. In that case we can release the semaphore and return back STOP. Hope this works.

@shankar-in it's hard to tell without all the code. Please create a draft PR for further discussion. Basically you have to store the stop state in an atomic and always return with STOP if the flag is set.
Also, could you please name the class just Timer? This makes it easier to extend it with methods like singleShot which wouldn't be periodic but needs manual restarting.

@elBoberido Okay I will raise the draft PR

@shankar-in just for information, since we are targeting Thursday for our 1.0RC1 release. It might take some time until non release critical PRs are reviewed.

@elBoberido  could you pls reassign this issue to my personal account?

thanks @dkroenke


control-service: job-builder using kaniko fix

The way we are preparing our job-builder image is like this:
FROM gcr.io/kaniko-project/executor

FROM alpine

COPY --from=0 /kaniko /kaniko

This is apparently known issue and caused this outage described in #2391
https://github.com/GoogleContainerTools/kaniko#known-issues

Running kaniko in any Docker image other than the official kaniko
image is not supported (ie YMMV).
This includes copying the kaniko executables from the official image
into another image.

So instead making sure we built from kaniko image directly should fix the issue
See https://github.com/vmware/versatile-data-kit/issues/2391

Good job!


Replace location dialog subtitle copy in requests from our SERP

Task/Issue URL: https://app.asana.com/0/1174433894299346/1200197271421338
Description:
We would like to replace the text in the location dialog in our DDG Android app with the copy that was in the black notification tooltip that appears for other browsers.
Screenshots:

duckduckgo.com
other sites

Steps to test this PR:
For duckduckgo.com

Open app and search for restaurants near me
Enable Location Services
Allow system location permission
The site permission dialog should now be shown
Check that the message is "We only use your anonymous location to deliver better results, closer to you. You can always change your mind later."

*For other sites

Open app and navigate to maps.google.com
Ask for current position
Allow system location permission
The site permission dialog should now be shown
Check that the message is "You can manage the location access permissions you’ve granted to individual sites in Settings."

Internal references:
Software Engineering Expectations
Technical Design Template

Thanks for the review @joshliebe!


Adding OpenTelemetry spans on the Go client's methods, CRI, GC and HTTP calls

What is the problem you're trying to solve
containerd 1.6 will have OpenTelemetry spans, but they are added automatically around gRPC calls. There are a few areas we should have spans to make troubleshooting easier.

The Go client's methods
CRI
GC operations
HTTP calls around container image operations

Describe the solution you'd like
We need to first define our naming conversion based on https://github.com/open-telemetry/opentelemetry-specification/tree/main/specification/trace/semantic_conventions. After that we can update each component accordingly.
Additional context
No response

Wahoo! My 2c is that spans should be on every entry/exit for tracing/debugging. So for example, CRI is all gRPC entry methods, it will automatically start a span. It does not need create internal sub-spans, (unless it wants to for some reason) but when it calls out to containerd via client (a new public API), or the task, or a windows/linux syscall, etc it may want to wrap that call in a span to show it's exit from the current code and entry into the next. This allows for entry and exit measurements, profiling, and tracing. Extremely helpful for both optimizing code, but also finding those hard to reach race conditions

Maybe another example since CRI is a "special" plugin. I think this idea holds for all of our services. Ex: We have a service api/local pattern. The span should be created at the service scope, once resolved to the local implementation we dont need a sub-span (most likely) but it is helpful for that service itself to have crated the child. This easily allows us to know when we have entered the "snapshotter service" etc. to do an operation and that it took 15 seconds.

add 1.7 milestone here cc @estesp

@kzys Hi, can I work on this issue.

@lengrongfu I did some work on this already and can have a PR ready by next week. I think we can reduce some overlap here since I am mainly focusing on CRI and related go client methods. Do you have any particular area of interest or if you can wait for my PR, that will be even better.

@lengrongfu I did some work on this already and can have a PR ready by next week. I think we can reduce some overlap here since I am mainly focusing on CRI and related go client methods. Do you have any particular area of interest or if you can wait for my PR, that will be even better.

I can pay attention to the point of "HTTP calls around container image operations", mainly where it interacts with the registry. There are also interactions with the registry in the CRI. Have you already dealt with it?

I can pay attention to the point of "HTTP calls around container image operations", mainly where it interacts with the registry. There are also interactions with the registry in the CRI. Have you already dealt with it?

Sure, I have not looked into that yet. Also you can take a look at GC operations as well.

@swagatbora90 Hi, i will be working around The Go client's methods, will this work be repeated?

@swagatbora90 Hi, i will be working around The Go client's methods, will this work be repeated?

HI @lengrongfu yes I am making changes to CRI runtime apis and and also adding spans in the related client methods which mainly in client.go, task.go, process.goand container.go.

I am listing out some of the current and planned items for adding otel spans to Containerd. This should help streamline some of the work, and others can contribute as well.
Current

[ ] GC operation - #7448
[ ] CRI Image service - #7453
[ ] CRI Runtime service - Pod sandbox APIs #7616
[ ] Containerd Go client:

[ ] client.go (7616)
[ ] container.go (7616)
[ ] process.go (7616)
[ ] task.go (7616)
[ ] pull.go (7453)

[ ] HTTP calls around container image operations - (7453)

TODO

[ ] CRI Runtime service - Container APIs
[ ] CRI stream server
[ ] Add Unit/Integration test #7493

@swagatbora90  I can look into the Add Unit/Integration test TODO task if you don't mind.


Made mamba-download the default and removed the flag from CLI

@wolfv are you ok to merge this?

I am still scared

ok let's do it :)

insert your key, I insert mine, then confirm your authentication code to the Central Station and I'll do the same!


Missing Parameter

When i run get-csTeamsMeetingPolicy, I get an AllowPSTNUsersToBypassLobby value but this is not documented above. Please add this

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 18b9baa8-da13-a24f-af31-91d63de13f08
Version Independent ID: 6a37be44-d957-2b7a-a563-53332cf59098
Content: Set-CsTeamsMeetingPolicy (skype)
Content Source: skype/skype-ps/skype/Set-CsTeamsMeetingPolicy.md
Service: skypeforbusiness-powershell
GitHub Login: @kenwith
Microsoft Alias: kenwith

@officedocsbot assign @yogkumgit

@kenwith  Please merge PR #3396

@dariomws306  Thank you very much for the contribution and updating the documentation with PR. @DeanGross Hope this update is helpful for you. Thanks for taking out some time to open the issue. Appreciate and encourage you to do the same in future also.

@officedocsbot close


Review ESLint vs. JSHint

Per this PR #82, review the additions and stylistic changes. May be worth pairing down the list of linting options down to the essentials and adding others to documentation w/ annotation on what they do.

Decision has been made to use ESLint moving forward but with a file custom to TMC preferences. Should be of little consequence if anyone wants to swap with their own .eslintrc.


Consider using 'organizations' for internal Gitea component

Right now, zarf pushes all package repositories to a Gitea user and then uses the Gitea API to add a read-only user to the new repo. This results in making an API call to Gitea for each new repo. If we were to create a Gitea Organization, and push our images to there, we could add the read-only user to the organization that way we don't have to make those API calls for every pushed repository.
This was discussed here which eventually rolled over to #666.

Closing since this duplicates #662