CKL Exports with Duplicate PDI IDs

We never did open a bug for this.  We have 3 hosts with the McAfee Virus Scan Local Client STIG applied, export an eChecklist, fill it out, import it, and then export ckls, and the ckls have several duplicate PDIs.  Chasing them down, they are all PDIs where there is the same STIG ID with another entry in the GoldDisk Table from a similar checklist (McAfee Managed Client STIG).  Oddly, with only the Local Client STIG attached, Sagacity also exported a ckl for the Manage Client STIG.
Importing the following eChecklist should reproduce the results.  If you check, there are a several PDIs that are duplicated in the ckl.
Matt has another bug in the same checklist I haven't yet been able to reproduce - where two Not a Finding PDIs (DTAM162, 163)get changed to Not Reviewed.  I'll post another bug for that when I can reproduce.
Win_7_-McAfee-original-eChecklist-1.xlsx

So it looks like one issue is that when I add the McAfee VS Local Client STIG for some reason it adds the Managed Client STIG as a Orphan.  That is probably one reason why the STIGs are being duplicated.  Couple troubleshooting steps...

On a fresh target (only OS assigned), add the McAfee VS Local Client STIG.
Export the target to an eChecklist
Verify the Orphan worksheet appears and delete it
Change statuses to some of the McAfee STIGs
Reimport the eChecklist
Review target to see if the Managed Client STIG was added
Export the target CKL file and see if the data is incorrect.

The secondary problem that Matt was probably running into was probably because the STIG status was being overwritten by the Managed Client STIG.
If what I think is happening is happening, then I think we should push this off to the next release as this would be cleared by up rewriting the eChecklist export methods to not export the Orphan findings.  If this HAS to be handled, then it is going to take me a while to track it down because this is getting into very complicated territory with the SQL queries.  In splitting the export functionality, it will allow me to simplify the queries quite a bit.

I'm afraid it has to be fixed.  It's both changing random findings and
creating unusable ckl files.
Why are the PDIs for the managed client STIG being added?  It seems like
that is the root of the problem - not the orphan checklist itself.
On Wed, Dec 5, 2018 at 12:48 PM Ryan P<EMAIL_ADDRESS>wrote:

So it looks like one issue is that when I add the McAfee VS Local Client
STIG for some reason it adds the Managed Client STIG as a Orphan. That is
probably one reason why the STIGs are being duplicated. Couple
troubleshooting steps...

On a fresh target (only OS assigned), add the McAfee VS Local Client
STIG.
Export the target to an eChecklist
Verify the Orphan worksheet appears and delete it
Change statuses to some of the McAfee STIGs
Reimport the eChecklist
Review target to see if the Managed Client STIG was added
Export the target CKL file and see if the data is incorrect.

The secondary problem that Matt was probably running into was probably
because the STIG status was being overwritten by the Managed Client STIG.
If what I think is happening is happening, then I think we should push
this off to the next release as this would be cleared by up rewriting the
eChecklist export methods to not export the Orphan findings. If this HAS to
be handled, then it is going to take me a while to track it down because
this is getting into very complicated territory with the SQL queries. In
splitting the export functionality, it will allow me to simplify the
queries quite a bit.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cyberperspectives/sagacity/issues/78#issuecomment-444620613,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aoyyh94RFn-7mViJnViHcJcaTAQALKRsks5u2CMhgaJpZM4ZDc1f
.

--

Jeff

Jeff Odegard
<EMAIL_ADDRESS>[image: LinkedIn] https://www.linkedin.com/in/jeffodegard/ [image:
Fakebook] https://www.facebook.com/jeff.odegard.98 [image: YouTube]
https://www.youtube.com/user/OdegardOnline

Yes, that is the root of the problem and that is what I'm trying to get you to test to see if we have a work around.  If deleting the orphan worksheet before reimporting fixes the issue as far as creating duplicate STIGs or overwriting statuses, then we can work with that until I can separate the orphan export.

After talking with Matt, I agree that we should just suppress exporting or
importing the orphan checklist for now.
On Wed, Dec 5, 2018 at 1:24 PM Ryan P<EMAIL_ADDRESS>wrote:

Yes, that is the root of the problem and that is what I'm trying to get
you to test to see if we have a work around. If deleting the orphan
worksheet before reimporting fixes the issue as far as creating duplicate
STIGs or overwriting statuses, then we can work with that until I can
separate the orphan export.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/cyberperspectives/sagacity/issues/78#issuecomment-444633427,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aoyyh5um-rJpdH4-pszWiUr-IEwxUhY3ks5u2CuHgaJpZM4ZDc1f
.

--

Jeff

Jeff Odegard
<EMAIL_ADDRESS>[image: LinkedIn] https://www.linkedin.com/in/jeffodegard/ [image:
Fakebook] https://www.facebook.com/jeff.odegard.98 [image: YouTube]
https://www.youtube.com/user/OdegardOnline

On a fresh target (only OS assigned), add the McAfee VS Local Client STIG.

created a fresh host, "foo" with Windows 10. - only the OS STIG and MS Firewall assigned
added the Mcafee Local Client - only that one applied.

Export the target to an eChecklist

done

Verify the Orphan worksheet appears and delete it

orphan worksheet contained  the checks for the Managed client STIG, which shouldn't have been added
deleted the orphan tab

Change statuses to some of the McAfee STIGs

done

Reimport the eChecklist

done (attached)
test-eChecklist-1.xlsx

Review target to see if the Managed Client STIG was added

it was not added.

Export the target CKL file and see if the data is incorrect.

only three STIGS (Win 10, MSFW, and McAfee Local Client were exported

To test our theory that it's when new hosts are created on eChecklist import, I modified the eChecklist so all the hostnames were foobar, then imported it.

only three STIGS were applied.  On ckl export, only the three correct STIGs' ckls were exported.

I'm still seeing duplicates.  I imported the two eChecklists in the attached zip file and then exported ckls, I ran my de-duplication script and had the script output found in the output.txt file.  The broken folder has all the original ckl's with duplicates.
ckl-dups-21Dec18.zip


Use latin1 for text encoding?

Hi folks!
First of all thanks for all efforts put into this project!
I have some schematics where accented characters were present in the texts and got some exceptions:
Traceback (most recent call last):
  File "altium.py", line 1615, in <module>
    main()
  File "altium.py", line 420, in main
    render(args.file, renderer.Renderer)
  File "altium.py", line 590, in __init__
    self.handle_children([objects])
  File "altium.py", line 627, in handle_children
    handler(self, owners, obj)
  File "altium.py", line 996, in handle_text_frame
    text=obj["TEXT"].decode("utf-8").replace("~1", "\n"),
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfc in position 5: invalid start byte

The problematic text was the following:
b'1x5 t\xfcskesor~190\xb0, 1,27mm'
Which corresponds to:
1x5 tüskesor\n90°, 1,27mm
I will do some experiments to map all the accented and special characters, but I am under an impression that Altium uses latin1 character encoding rather than plain ASCII.

I expect it uses something like Latin-1 or Windows-1252. I am happy to change line 996 to decode with Latin-1. However I noted under https://github.com/vadmium/python-altium/blob/master/format.md#pin that I saw the byte 0x8E representing a broken bar (U+00A6, ¦). So the full story might not be so simple.
I have come across parallel UTF-8 properties, for instance as well as one named TEXT, there is one named %UTF8%TEXT. You don’t know if your text frame object has a UTF-8 version of the text?

Hi @vadmium
I have not found any occurrence of the "UTF" string in the file.
I think I will create a text with including the most accents, and special characters, save it and see the text to make more solid conclusion on the encoding type.


Cannot Get Both Cameras to Work

I'm testing the program while I wait for the parts to make the headset.  I am concerned that I can't get it to work with the cameras I have.
I have a PS3 Eye camera which works with a CL Eye driver through ManyCams.  The HP webcam built in and another USB camera both do not work.  When I open the Microsoft camera app, both cameras DO work; so I don't think it is a driver issue.
Any ideas on what is wrong and how to fix it?  I'm not a programmer but am really interested in the project as I want to track student eye movement as they read.  Please help.

I have the same issue currently. I can run the code and access the camera viewer, but every time I attempt to select a USB camera or Integrated camera option and click 'go' I receive a message that my camera is being used in another process. I have reset and fiddled with my camera settings several times. I'd also appreciate some assistance-

did you find a solution for this? i would really apprecitate help or a possible solution


Fix multi-arch container

Move build steps into container to actually build for other architectures. Fixes arm64 support added in #298.

@barthalion pull_request_target uses the workflow file from main which still has the step to build flatpak-builder outside of the Dockerfile. But as the new Dockerfile doesn't copy these files in, this could actually work.

@barthalion Also as the event is not pull_request but pull_request_target this will actually push the image with tag latest.

bwrap: Creating new namespace failed: Operation not permitted, I guess we need to run the container as --privileged or run flatpak-builder with --disable-rofiles-fuse?

flatpak-github-actions uses --privileged: https://github.com/flatpak/flatpak-github-actions/blob/b8be6bc0ede43cb3a2cc55895654312b14f84a0d/README.md?plain=1#L31. I will try with --disable-rofiles-fuse.

They use it to run the image, not build it.

But it is the image running flatpak-builder...

Did I disable fuse correctly?
I get
#20 104.4 FB: Running: flatpak build --die-with-parent --env=FLATPAK_BUILDER_BUILDDIR=/run/build/libxcvt --nofilesystem=host:reset --filesystem=/build/org.flatpak.Builder/.flatpak-builder/build/libxcvt-1 --bind-mount=/run/build/libxcvt=/build/org.flatpak.Builder/.flatpak-builder/build/libxcvt-1 --build-dir=/run/build/libxcvt/_flatpak_build --bind-mount=/run/ccache=/build/org.flatpak.Builder/.flatpak-builder/ccache --env=SOURCE_DATE_EPOCH=1708294716 '--env=CFLAGS=-O2 -pipe -g -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer ' '--env=CXXFLAGS=-O2 -pipe -g -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fexceptions -fstack-protector-strong -grecord-gcc-switches -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer ' '--env=LDFLAGS=-L/app/lib -Wl,-z,relro,-z,now -Wl,--as-needed ' --env=V=1 --env=MOUNT_FUSE_PATH=../tmp/
#20 104.4 bwrap: Creating new namespace failed: Operation not permitted

Seems to work with security.insecure which the BuildKit's equivalent to --privileged (see last commit).

With security.insecure, I now get Creating new namespace failed: Invalid argument.

It doesn't seem to be possible to create manifests for images which are not yet pushed. I created the steps as I think they work but can't test them without permissions to push to a registry. @barthalion could you please take over from here?

@barthalion Thank you so much for your help! Works like a charm!


Command Permissions

Add a property to the Command class to lock it behind a permission with Discord's native permission settings in integrations.

Ima give this a try


download file with dest property that does not exist causes file to be downloaded to root.

Is there an option to create a destination path where one does not exist?

Yes, this is possible. Just create the directory in a doFirst block before downloading:
task downloadFile(type: Download) {
    def destDir = "mydirectory"
    doFirst { mkdir destDir}
    src 'http://example.com/index.html'
    dest destDir
}

Explanation:
gradle-download-task only creates a destination directory if it equals the build directory or if multiple source files should be downloaded. Otherwise, the dest property defines the name of the destination file. See also lines 139-152 in DownloadAction.java.
https://github.com/michel-kraemer/gradle-download-task/blob/b3f492521d6999ad83d83dc871c2785132856bca/src/main/java/de/undercouch/gradle/tasks/download/DownloadAction.java#L139-L152

Thank you !
On Sat, Jul 28, 2018 at 1:36 AM, Michel Krämer<EMAIL_ADDRESS>wrote:

Yes, this is possible. Just create the directory in a doFirst block
before downloading:
task downloadFile(type: Download) {
def destDir = "mydirectory"
doFirst { mkdir destDir}
src 'http://example.com/index.html'
dest destDir
}
Explanation:
gradle-download-task only creates a destination directory if it equals the
build directory or if multiple source files should be downloaded.
Otherwise, the dest property defines the name of the destination file.
See also lines 139-152 in DownloadAction.java.
https://github.com/michel-kraemer/gradle-download-task/blob/
b3f492521d6999ad83d83dc871c2785132856bca/src/main/java/de/
undercouch/gradle/tasks/download/DownloadAction.java#L139-L152
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/michel-kraemer/gradle-download-task/issues/116#issuecomment-408592565,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AeFrB2qw29bwRThr0XSha8AMvF_qU_Ntks5uLCKNgaJpZM4VbvZ7
.


Vector2 and Vector3 are not properly parsed

This commit should solve https://github.com/godotengine/godot/issues/13425
It has been tested manually and it works like a charm

This 2,1 branch right? I dont remember any vector2 setting being saved
there but if it fixes it, seems fine
On Fri, Dec 1, 2017 at 6:31 AM, Rémi Verschelde<EMAIL_ADDRESS>wrote:

@akien-mga https://github.com/akien-mga requested your review on:
godotengine/godot#13433 https://github.com/godotengine/godot/pull/13433
Vector2 and Vector3 are not properly parsed.
—
You are receiving this because your review was requested.
Reply to this email directly, view it on GitHub
https://github.com/godotengine/godot/pull/13433#event-1367297895, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AF-Z233R1gb-_KblwMRKALX_yrKY_Mn_ks5s78dvgaJpZM4QxzFT
.

In fact there are 3 in the project settings:
display > custom mouse cursor hotspot
physics 2d > default gravity vector
physics > default gravity vector


More details about server timing sections

Hi
I just used your plugin to diagnose server response time issue.
In my case, template processing is the most problematic area (2.9 s of total 4s).
Could you elaborate more about this section?
How can I reduce the time spend in this section? Just replace the theme? is there any other solutions?
Thank you.

Hi @aminjabari!
Usually it means that you did not separated the business logic from the presentation and in your (for example front-page.php) template file. So you do all the db related logic and filtering the data there instead of functions.php.
What you can do is check the db queries: maybe they can be simplified with a plain sql query.
Check for sorting and filtering logic in the php: most of the time, they can be moved to the db query.
Check for file related activities: some themes load SVG images via file_get_contents which results in increased server response time.
Also you can use WP Rocket or other caching plugin wich will basically zero out this section, because they will serve pre-generated static HTML files instead of processing the template in real time.


[WIP] Add union remote

This PR adds support for a new remote: union.
The union remote "merges" multiple remotes into one logical structure. The content of directories with the same name are merged into a single directory. The ordering of the remotes decides which remote "overwrites" others if some files in a directory have the same name. The next remote in the list takes precedence. The last remote is used to write files, all other remotes are read-only.
Closes #2467

This is looking nearly complete :-)  How do you think you are doing?
It looks like you'll need to run goimports on the files to fix the formatting of the includes as it does it slightly differently to go fmt.

Not sure if i did that correctly, i configured goimports style in goland
What about the integration test?
2 issues i see:

one failure because i'm wrapping multiple fs and thus the returned fs is different from mine
currently using (*local.Object) as Nil object for the test setup because my union fs test remote wraps three local fs's. If i just input Nil it can't find the local fs backend during the tests

I've managed to fix the integration test with some fixes to the test framework.
I've squashed your commits and added one of my own which fixes the fs and sync integration tests and pushed these to the union branch.
https://github.com/ncw/rclone/tree/union
https://github.com/ncw/rclone/commits/union
Can you take a look over that and see if you are happy with it?
I think there are a few doc bits which need to be done, but otherwise it is looking complete :-)
https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#writing-a-new-backend
Do you want to send a PR (on top of the union branch) for those last doc bits? I'm happy to do it if you haven't got time.
Thanks
Nick

Thanks, LGTM!
The only thing left regarding docs i found was the missing "union.md" in the make_manual.py, i ignored all other places which only list real filesystems and do not include alias as well as union isn't a "real" cloud storage provider but an abstraction. Did i miss some place regarding docs? In either case feel free to add the missing parts if necessary, greatly appreciated!

Great!
I'l give it the once over and double check missing bits then merge :-)

Nice thanks!

I've merged that now - thank you very much indeed for your contribution - very useful :-)
9e3ea3c6acbf1f427d82914ac8cc4ef91281ea8d

Nice, thanks!


Update_attributes_analysis_methods

Fixes issue (if applicable) #
Proposed Changes

Updated the attribute slot to the Analysis Method

Checks

[ ] Builds locally

Update addressed in #169


[FLINK-12184] history server compatible with old version

What is the purpose of the change
This PR fixes FLINK-12184. The problem is that the HistoryServerArchiveFetcher can't convert legacy job overview.
Get the "scheduled" field instead of getting the "pending" filed which doesn't exist in legacy job overview json.
Brief change log

The tasks get "scheduled" field instead of getting "pending" field.

Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): (no)
The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
The serializers: (no)
The runtime per-record code paths (performance sensitive): (no)
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
The S3 file system connector: (no)

Documentation

Does this pull request introduce a new feature? (no)

@yumengz5 do you still want to work on this fix? Otherwise, @klion26 might wanna take over?

@tillrohrmann  thanks for your remind. @yumengz5 if you do not mind, I can take over this.

@klion26 Sorry about the late reply. All right. You can take over it.

@yumengz5 thanks for your reply, I'll file a new pr for this issue.

All right, then I'll close this PR @yumengz5 since @klion26 will open a new PR.


Multi-page format - support multiple pages

In recent user testing, the single page format didn't always test well. With further testing, it could be beneficial to support multiple pages. As an example, see the GOV.UK developers or GDS Way implementation of the template.

@soupdragon99 guessing this is about bringing in the work that GOV.UK did? @tijmenb is that something you've been working on?

multi-page format  a priority for Registers - Phil already set this up for The GDS Way, how translatable is this to wider audience? PaaS are also want this functionality for their docs.

Verify also need this functionality.

https://github.com/alphagov/tech-docs-template/pull/143 adds some docs to the README of the template to show how to create a separate page. No code changes are needed to support that.


Support Uber's H3 geospatial indexing system as geohex_grid

Description
This feature introduces new grid system (geohex_grid) for aggregation based on geo_point.

[x] Add H3 as native java library
[x] Add new aggregation context
[x] Add integration test

Issues Resolved
#98
Check List

[x] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Codecov Report

Merging #179 (9c6c524) into main (4465452) will increase coverage by 0.51%.
The diff coverage is 88.04%.

@@             Coverage Diff              @@
##               main     #179      +/-   ##
============================================
+ Coverage     84.64%   85.16%   +0.51%
- Complexity      332      377      +45
============================================
  Files            44       52       +8
  Lines          1094     1186      +92
  Branches         97       98       +1
============================================
+ Hits            926     1010      +84
- Misses          128      135       +7
- Partials         40       41       +1

Impacted Files
Coverage Δ

...arch/aggregations/bucket/geogrid/GeoHexHelper.java
75.00% <75.00%> (ø)

...s/bucket/geogrid/GeoHexGridAggregationBuilder.java
76.00% <76.00%> (ø)

...search/aggregations/bucket/geogrid/GeoHexGrid.java
90.90% <90.90%> (ø)

...opensearch/geospatial/plugin/GeospatialPlugin.java
92.85% <100.00%> (+1.94%)
:arrow_up:

...regations/bucket/geogrid/GeoHexGridAggregator.java
100.00% <100.00%> (ø)

...ns/bucket/geogrid/GeoHexGridAggregatorFactory.java
100.00% <100.00%> (ø)

.../aggregations/bucket/geogrid/GeoHexGridBucket.java
100.00% <100.00%> (ø)

.../aggregations/bucket/geogrid/ParsedGeoHexGrid.java
100.00% <100.00%> (ø)

...gations/bucket/geogrid/ParsedGeoHexGridBucket.java
100.00% <100.00%> (ø)

...earch/geospatial/index/mapper/xypoint/XYPoint.java
74.13% <0.00%> (+5.17%)
:arrow_up:

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Approving. Please add all the related PR in the description of this PR. So that we know all the related PRs which got approved.

Ack


docs: remove dead badge

coverage: 100.0%. remained the same
when pulling 5e022464d6de93937dc708f56237a677bc22eacc on devpow112-patch-1
into 6bd64e9fb33a42951ad508acb87c428ea35639c9 on main.

:tada: This PR is included in version 2.0.3 :tada:
The release is available on:

GitHub release
npm package (@latest dist-tag)

Your semantic-release bot :package::rocket:


problem at build

Hi,
It seems the build is failing. I've run docker build -t=sverhoeven/cartodb docker-cartodb/ and I get the following error at step 23/49:
RUN service postgresql start && /bin/su postgres -c       /tmp/template_postgis.sh && service postgresql stop
 ---> Running in 67a91db53068
 * Starting PostgreSQL 10 database server
   ...done.
bash: /tmp/template_postgis.sh: /bin/bash^M: bad interpreter: No such file or directory
The command '/bin/sh -c service postgresql start && /bin/su postgres -c       /tmp/template_postgis.sh && service postgresql stop' returned a non-zero code: 126

Hmm, I am unable to reproduce this error. It builds correctly on my local machine and on Docker Hub.
Step 23/49 : RUN service postgresql start && /bin/su postgres -c       /tmp/template_postgis.sh && service postgresql stop

 ---> Running in 71571d79a9f8

 * Starting PostgreSQL 10 database server

   ...done.

Error:
pg_wrapper: createlang was not found in /usr/lib/postgresql/10/bin

UPDATE 1

Are you using a up to date clone of https://github.com/sverhoeven/docker-cartodb.git and latest version of ubuntu:18.04 Docker image?
Also try to build without cache using docker build -t=sverhoeven/cartodb --no-cache docker-cartodb/

Thank you for your quick response. Weird! I believe I did everything you suggested, but get the exact same error. I'm new to Docker, there might be something obvious I'm missing.
docker pull ubuntu
git clone https://github.com/sverhoeven/docker-cartodb.git
docker build -t=sverhoeven/cartodb --no-cache docker-cartodb/

sverhoefen and rferrali.
Found the solution thanks to webodm project.
When using the git clone command append it with the following at the end of the line:
--config core.autocrlf=input
There is a difference of how git on Windows and the the nix'e (Linux and OSX) handle linefeeds at the end of text files etc.
I am running Windows.
sverhoefen
Could you please update the docs with this.  Could you also please update the docs so that for Windows users one has to use the docker ip address, usually <IP_ADDRESS> in stead of <IP_ADDRESS>.
Hope this helps.
Pieter

Thank you Pieter, that solved my problem.


Create MongoDB dashboards & alerts

The MongoDB dashboards are currently empty and there is no job/package for MongoDB alerts.

any updates can U add https://grafana.com/dashboards/2583

noneed if  grafana on postgres db store


Update lombok jar to 1.18.10

Update to version 1.18.10 for new features and bug fixes.

Thanks! I will publish it tonight

Published to VSCode marketplace


cant connect to my raspberry pi w

[2022-07-04 11:16:45] Using "/data/options.json" config file
[2022-07-04 11:16:45] MQTT Host defined in config file. Ignoring Supervisor Configuration...
[2022-07-04 11:16:45] RTLAMR2MQTT Starting...
[2022-07-04 11:16:45] Using an external RTL_TCP session at <IP_ADDRESS>:1234
[2022-07-04 11:16:45] Configured MQTT sender:
[2022-07-04 11:16:45]  > hostname => <IP_ADDRESS>
[2022-07-04 11:16:45]  > port => 1883
[2022-07-04 11:16:45]  > username => mqtt
[2022-07-04 11:16:45]  > client_id => rtlamr2mqtt
[2022-07-04 11:16:45]  > base_topic => rtlamr
[2022-07-04 11:16:45]  > availability_topic => rtlamr/status
[2022-07-04 11:16:45]  > tls => None
[2022-07-04 11:16:45] Sending message to MQTT:
[2022-07-04 11:16:45]  > topic => rtlamr/status
[2022-07-04 11:16:45]  > payload => online
[2022-07-04 11:16:45]  > retain => True
[2022-07-04 11:16:45] Trying to start RTLAMR: /usr/bin/rtlamr -msgtype=scm+ -format=json -filterid=62331584 -unique=true -centerfreq=915000000 -symbollength=32 -tunergain=40 -server=<IP_ADDRESS>:1234
[2022-07-04 11:16:45] RTLAMR started with PID 10
[2022-07-04 11:16:45] 11:16:45.897614 decode.go:45: CenterFreq: 915000000
[2022-07-04 11:16:45] 11:16:45.897735 decode.go:46: SampleRate: 1048576
[2022-07-04 11:16:45] 11:16:45.897740 decode.go:47: DataRate: 32768
[2022-07-04 11:16:45] 11:16:45.897742 decode.go:48: ChipLength: 32
[2022-07-04 11:16:45] 11:16:45.897745 decode.go:49: PreambleSymbols: 16
[2022-07-04 11:16:45] 11:16:45.897747 decode.go:50: PreambleLength: 1024
[2022-07-04 11:16:45] 11:16:45.897749 decode.go:51: PacketSymbols: 128
[2022-07-04 11:16:45] 11:16:45.897751 decode.go:52: PacketLength: 8192
[2022-07-04 11:16:45] 11:16:45.897754 decode.go:59: Protocols: scm+
[2022-07-04 11:16:45] 11:16:45.897756 decode.go:60: Preambles:<PHONE_NUMBER>100011
[2022-07-04 11:16:45] 11:16:45.897759 main.go:124: GainCount: 14
[2022-07-04 11:16:50] 11:16:50.901222 main.go:343: Receiver context cancelled.
[2022-07-04 11:16:50] 11:16:50.901468 main.go:320: read tcp <IP_ADDRESS>:37042-><IP_ADDRESS>:1234: i/o timeout
[2022-07-04 11:16:50] io.ReadFull
[2022-07-04 11:16:50] main.(*Receiver).Run.func2
[2022-07-04 11:16:50] 	/go/pkg/mod/github.com/bemasher/rtlamr@v0.9.3/main.go:181
[2022-07-04 11:16:50] runtime.goexit
[2022-07-04 11:16:50] 	/usr/local/go/src/runtime/asm_amd64.s:1571
[2022-07-04 11:16:50] Sleep_for defined, time to sleep!
[2022-07-04 11:16:50] Terminating all subprocess...
[2022-07-04 11:16:50] Kill process called.
[2022-07-04 11:16:50] Killing RTLAMR...
[2022-07-04 11:16:50] Killed in the first attempt.
[2022-07-04 11:16:50] Sleeping for 300 seconds, see you later...
my rtl-sdr on my raspberry pi w, keeps saying Use the device argument 'rtl_tcp=<IP_ADDRESS>:1234' in OsmoSDR.... what does this mean?

I got it working by using my dell workstation and unraid
does this mean its working?
[2022-07-04 13:51:52] Using "/data/options.json" config file
[2022-07-04 13:51:52] MQTT Host defined in config file. Ignoring Supervisor Configuration...
[2022-07-04 13:51:52] RTLAMR2MQTT Starting...
[2022-07-04 13:51:52] RTL SDR Device 0bda:2838 found on USB port 001:003 - Index: 0
[2022-07-04 13:51:52] No USB device specified in the config file, using the first found.
[2022-07-04 13:51:52] Configured MQTT sender:
[2022-07-04 13:51:52]  > hostname => <IP_ADDRESS>
[2022-07-04 13:51:52]  > port => 1883
[2022-07-04 13:51:52]  > username => mqtt
[2022-07-04 13:51:52]  > client_id => rtlamr2mqtt
[2022-07-04 13:51:52]  > base_topic => rtlamr
[2022-07-04 13:51:52]  > availability_topic => rtlamr/status
[2022-07-04 13:51:52]  > tls => None
[2022-07-04 13:51:52] Reseting USB device: /dev/bus/usb/001/003
[2022-07-04 13:51:53] Reset sucessful.
[2022-07-04 13:51:53] Sending message to MQTT:
[2022-07-04 13:51:53]  > topic => rtlamr/status
[2022-07-04 13:51:53]  > payload => online
[2022-07-04 13:51:53]  > retain => True
[2022-07-04 13:51:58] Unknown exception connecting to MQTT broker: timed out
[2022-07-04 13:51:58] Trying to start RTL_TCP: /usr/bin/rtl_tcp  -s 2048000
[2022-07-04 13:51:58] RTL_TCP started with PID 10
[2022-07-04 13:52:03] RTL_TCP is ready to receive connections!
[2022-07-04 13:52:03] Trying to start RTLAMR: /usr/bin/rtlamr -msgtype=scm+ -format=json -filterid=62331584 -msgtype=all
[2022-07-04 13:52:03] RTLAMR started with PID 15
[2022-07-04 13:52:03] 13:52:03.240469 decode.go:45: CenterFreq: 912380000
[2022-07-04 13:52:03] 13:52:03.240628 decode.go:46: SampleRate: 2359296
[2022-07-04 13:52:03] 13:52:03.240637 decode.go:47: DataRate: 32768
[2022-07-04 13:52:03] 13:52:03.240642 decode.go:48: ChipLength: 72
[2022-07-04 13:52:03] 13:52:03.240647 decode.go:49: PreambleSymbols: 32
[2022-07-04 13:52:03] 13:52:03.240652 decode.go:50: PreambleLength: 4608
[2022-07-04 13:52:03] 13:52:03.240657 decode.go:51: PacketSymbols: 736
[2022-07-04 13:52:03] 13:52:03.240661 decode.go:52: PacketLength: 105984
[2022-07-04 13:52:03] 13:52:03.240668 decode.go:59: Protocols: scm,scm+,idm,r900
[2022-07-04 13:52:03] 13:52:03.240674 decode.go:60: Preambles:<PHONE_NUMBER>01001100000,0001011010100011,01010101010101010001011010100011,00000000000000001110010101100100
[2022-07-04 13:52:03] 13:52:03.240686 main.go:124: GainCount: 14
also, how do i get it listening for everything???

sorry, does this listen on only the 900 mhz band? accorind go pge, their gas meters run on 450 MHZ, do i need to change a setting?
https://wireless2.fcc.gov/UlsEntry/attachments/attachmentViewRD.jsp;ATTACHMENTS=zL8vvDdQwZPQL60LyGd5hb042FhMCnJTlHbL6YyJRnpYLdPg0y2s!1236233192!523590052?applType=search&fileKey=373281438&attachmentKey=18619117&attachmentInd=applAttach


etcdctl/ctlv2/command: fix type switch case order

Since syscall.Errno implements net.Error and all cases
are matched sequentially, it's a mistake to put syscall.Errno
case after net.Error since it will never be executed.
This change swaps syscall.Errno case with net.Error
to give that clause chance to execute.
Found using https://go-critic.github.io/overview#caseOrder-ref

lgtm


Release 4.0.0 for Windows

Replace all (7) audio players for WORKPLACE BACKGROUND NOISE -- Office Sounds for Focusing While You Work Remotely with x1 Warm Pink Noise with 432hz Boost • 10 hours • Black Screen.

https://github.com/kkamara/kelvinkamara/releases


Customer log-in from the admin panel is redirected to the same tab.

Terms

[X] Before you openning this issue, i have checked if the issue has already been reported.

Bagisto Version(s) affected
master
Issue Description
Customer log-in from the admin panel is redirected to the same tab. it should be redirected to the new tab.
Preconditions

Admin should be logged in.
Admin should have at least one customer.

Steps to reproduce

Go to 'Admin panel'
Click on 'Customers'
Click on the login user option
See the error

Expected Result
It should be redirected to the new tab.
Actual Result
Redirected to the same tab. Here
Screenshots
No response
Additional context
No response

Now, this issue is fixed. so I am closing this issue.
And I have attached a screenshot Here, for future reference.


Refactor Sprite rotation function names

The rotation functions in the Sprite class have poor and potentially confusing names and arguments.
Refactoring as
Sprite
{
//...
    void rotation(float angleDegrees) const;
    float rotationDegrees() const;
    float rotationRadians() const;
//...
};

is much more clear and less confusing.

You should provide a bit more context on this. I assume you're talking about the Sprite class. This also relates to Renderer.

One concern I have here is the inability to overload based on angle type if they are both float. In your example, you have both rotationDegress and rotationRadians to account for that, while the other direction only accepts a single measure type.
Maybe what we really want is an Angle type, that can be initialized by factory functions for either measure, and convert to either measure. Then there would be no need for overloads to handle either measure type. You could pass Angle to any function that needs angles, and the measure of the angle would be specified unambiguously when the type is constructed.

Forcing the user to stick to one measurement unit, degrees or radians, from outside the class is less error prone. If the internals expect something else they can be converted.
For non-mathy types, like Sprite, specifying the angle in degrees is more clear. (Really, unless otherwise known and expected to be in radians like std::sin, etc., I would prefer every function take in an angle argument as Degrees; however, it's way more clear to name the variable angleDegrees anyway).
Mathy-types can have a degrees implementation for ease-of-use like so:
float CosDegrees(float degrees)
{
    const auto radians = MathUtils::ConvertDegreesToRadians(degrees);
    return std::cos(radians);
}

float SinDegrees(float degrees)
{
    const auto radians = MathUtils::ConvertDegreesToRadians(degrees);
    return std::sin(radians);
}

float Atan2Degrees(float y, float x)
{
    const auto radians = std::atan2(y, x);
    return MathUtils::ConvertRadiansToDegrees(radians);
}

I think my point may have been missed. An Angle is a concept that can be represented as either degrees or radians, and those two representations are equivalent. This is a bit like how a point can be specified as either Cartesian coordinates or polar coordinates. They can be freely converted between each other. You can have an Angle class that can be initialized from either degree or radian data, and to return values in either degree or radian form. This is like how a point class can be initialized in either Cartesian form or polar form, and return equivalent data in either form. The internal format doesn't matter from the outside in terms of correctness. It might only matter in terms of efficiency, and how often conversions are performed. An Angle might be most effectively represented internally with radians if it's being used a lot in formulas, much like how a point might be most efficiently stored in Cartesian coordinates when used for 2D raster images. Either can be displayed in an alternative form for user display, depending on the appropriateness of the context.
Example:
renderer.drawImageRotated(image, Angle::FromDegrees(90));
renderer.drawImageRotated(image, Angle::FromRadians(Math::Pi / 2));

Example:
float SinDegrees(Angle angle)
{
    return std::sin(angle.radians());
}

That seems to be adding undue complexity of an entire class where there doesn't need to be; as well as punting the conversion to the user where it could easily be used incorrectly. Just by looking at the examples, I couldn't tell you what Angle represents.

That's the point though. You don't need to know the internal representation of Angle. Users of Angle will know what measure they want, and request the correct type. Providers of angles will know what they have, and can create from the available type.
It also avoids having to create overloads for different angle types. Not that you could create overloads with either this nor the previous idea, since there is only one type, and hence no type distinction to provide an overload. I suppose I should say there's no need for near duplicate functions for every consumer of Angle values.

At any rate, the simpler solution is to just not worry about different angle measures and define only one correct type, and try to use that everywhere. Internally, I think radian measure makes the most sense. It's the natural measure of angles, and allows for the highest efficiency of use. For user input, degree makes sense, and can be converted on the fly when loading data, or when presenting data to the user.
What I don't want, is to provide multiple functions for every possible consumer of angles.

At any rate, the simpler solution is to just not worry about different angle measures and define only one correct type, and try to use that everywhere. Internally, I think radian measure makes the most sense. It's the natural measure of angles, and allows for the highest efficiency of use. For user input, degree makes sense, and can be converted on the fly when loading data, or when presenting data to the user.
What I don't want, is to provide multiple functions for every possible consumer of angles.

I totally agree. User-facing should be degrees (again, except for the trig functions, i.e. sin/cos/tan/atan because those are designed from C++ to be in radians). Internally we should decide on one and stick with it. Mathematically, radians would be better; but degrees are easier to understand.

I'm a little torn here as I'm not sure if I like the idea of an Angle class, though considering the direction we've taken with NAS2D so far, it would make sense. The user can specify their Angle in whatever unit they want and internally functions that use Angle would handle it in whatever way is most efficient.
At the moment all angle parameters are generally expected to be in degrees. That's was the easiest way for me to think about it so that's how the functions ultimately handle them.
Going with the example in the first post, I wonder if it would make sense to expect all functions/parameters in degrees but offer a stand-alone function that can convert to radians as needed? Now that I think about it those functions already exist in the Trig.h file. It's a bit crude but it suited my needs at the time.

I think that solution is still suitable, and is the one I'm leaning towards. Have just a single correct way to specify angles, and separate helper methods to convert between angle measures. No need to start adding overloads everywhere to handle either angle measure when it can just be a separate wrapped call. That's basically what we already have, and I'm fine with that.


Suppress and revive offers based on instance state

Summary:
Changed the ReviveOffersActor to react to instance state changes, instead of being triggered from within the launch queue. Every time an instance is scheduled, a reviveOffers call will be issued. Once all scheduled instances transitioned to another state, offers will be suppressed.
It is to be discussed if and how we want to handle revives when Marathon needs to unreserve/destroy. Also, we should not revive when scheduled instances are implicitly backed off.
JIRA issues: MARATHON-8627

Superseded by 6936


fix warnings, add pp boilerplate

Hi,
I fixed the warnings (before each commit, you should run dune build @all to make sure everything is fine).
I also added some boilerplate in pp.ml to help you get started.

Hi,
Yeah i've indeed committed in a hurry yesterday ! :x
Changes reviewed, thanks for your help !


Issues using certificate with AWS

I did a real-world test of our system today.  My goal was to do the following:

Run an nginx container on AWS via Elastic Container Service, and put a Load Balancer in front of it
Create a CNAME record on My.Custom.Domain that points to the Load Balancer's domain
Create an HTTPS certificate on My.Custom.Domain
Import this certificate into AWS Certificate Manager
Associate this certificate with the Load Balancer, and allow traffic on https://mycustomdomain...

I am pleased to report that I was able to accomplish all of this, but I encountered some issues.  Here is the process.

I created an nginx cluster with load balancer at http://ec2co-ecsel-1rzway2anzrs6-856795930.us-east-1.elb.amazonaws.com/
I created a CNAME record at aws.davidhumphrey.stage.mystudentproject.ca which points to ec2co-ecsel-1rzway2anzrs6-856795930.us-east-1.elb.amazonaws.com (you can test it at http://aws.davidhumphrey.stage.mystudentproject.ca/)
I created a certificate, and it worked well.  I also received an email, see #523

The next part was more problematic.  In Amazon Certificate Manager, you are presented with the following screen:

If I copy/paste our Public Key as the Certificate Body, and our Private Key as the Certificate Private Key, I get this:

AWS is not liking that we're using the certificate chain vs. only the certificate.  If I edit this PEM file to only include the first certificate, and delete the second and third in the chain, it works.
If I paste our "Public Key" into the "Certificate chain" box, it works.
I was then able to do step 5. and add a Listener on port 443 for the Load Balancer, as well as updating its Security Group for HTTPS traffic and now I can create a secure connection at https://aws.davidhumphrey.stage.mystudentproject.ca/
NOTE: this will prompt you with a warning, since we're using the Let's Encrypt Staging server, and it's not in the Browser's trusted list of CAs.  You can click "Advanced" and allow the connection (I promise you, I'm not trying to hack you :)
I think we need to figure out how we want to deal with this in our Certificate UI.  Given that the first time I tried to use our stuff, I was forced to manually hack this file, we should consider if we need a 3rd thing:

Public Certificate (only the first of the 3 in the full chain)
Private Key (same as now)
Full Chain (all 3 certificates in full chain)

Looking at node-acme, it seems we could call crypto.getPemBodyAsB64u() to get the public cert from the chain.  Maybe we need 3 colums in the cert table:

certificate
privkey
fullChain

I did some testing, and here is one solution using crypto.splitPemChain().  In app/queues/certificate/order-completer-worker.server.ts and completeOrder(), we can do this:
    this.#order = await this.#client.finalizeOrder(this.#order, csr);

    // the fullChain is the public cert + the intermediate certificate(s) for Let's Encrypt
    const fullChain = await this.#client.getCertificate(this.#order);
    // the cert is the public cert, chain is the intermediate certificate(s)s for Let's Encrypt
    const [cert, ...chain] = acme.crypto.splitPemChain(fullChain);

Now we the 4 things you might need:

public cert
private key
chain
full chain

I think we should either update the Certificate model code to split these things out at runtime, or alter the database to store them.  We should also expose all 4 in the UI.

@Eakam1007 has volunteered to take this on (thank you!).  I think we should:

Add 2 new columns to the Certificate table: chain (i.e. only the intermediate certificate(s)) and fullChain (the certificate +  chain together).  What we call certificate now can be the public cert only
Update the UI to add 2 more areas for the chain and fullChain
Modify completeOrder() like I mention above
Fix any other callers that will be affected by this change.


!!/unwatch suganthan

!!/unwatch suganthan - it's causing too many false positives (in usernames)

!!/approve


Generate Synthetic Radar Measurements

Simulated range/range-rate measurements can now be generated from SbgatCore and SbgatGui . Current work aims at streamlining results visualization from within SbgatGui, with the possibility to visualize images using VTK or saving them to a file/gif (when a sequence of images is collected).

Tasks to complete:

[ ] Normalize image values by largest value: contract is too small when only image is captured, and successive images are not consistent with one another
[ ] Add visualizer within SbgatGui
[ ] Add ImageMagick call to generate Gif from image sequence
[ ] Investigate ray-tracing to ensure consistency


errors : Unable to extract container id

Issue description

issue description: The SWCL node task report errors while running the  MNIST example. The SWOP report errors "Unable to extract container ID "
occurrence - consistent :
error messages: Unable to extract container ID

Swarm Learning Version: 1.00
OS and ML Platform

details of host OS: Ubuntu 22.04 LTS

Additional notes

Are you running documented example without any modification?   Only modify the IP in the  profile files under folder.

I believe the cgroup version is v2 in ubuntu 22.04.
That may have affected it.

How about the following as a tentative work-around?
I modified the run-swop script as follows
diff --git scripts/bin/run-swop scripts/bin/run-swop
index 1987e15..e8a30ba 100755
--- scripts/bin/run-swop
+++ scripts/bin/run-swop
@@ -97,6 +97,8 @@ processComponentBatchOpt()
         --swop-uid) checkAndAssign "${opt}" "${optarg}"
             re='^[0-9]+$'
             [[ ! "${optarg}" =~ ${re} ]] && error "${opt}: ${optarg}: bad user id";;
+        --cgroupns) checkAndAssign "${opt}" "${optarg}"
+            ;;
         *) unprocessedOpts+=("${origParam}"); nShift=1;;
     esac

@@ -126,6 +128,8 @@ onTrainEnd()
     envvar+=(-e "SWOP_GID=${dockerGroupId}")
     envvar+=(-e "SWOP_PROFILE=${profileFileName}")

+    [[ -n "${cgroupns}" ]] && miscDockerRunParams+=(--cgroupns ${cgroupns})
+
     cmd+=("${unprocessedOpts[@]}")
     unprocessedOpts=()

And, I added --cgroupns host to the command line option of the run-swop script and it works on Ubuntu 22.04.

Thank you @IMOKURI for work around, if this is working then it can be used by other users. We are working on the fix and should be available in our upcoming release.

How about the following as a tentative work-around?
I modified the run-swop script as follows
diff --git scripts/bin/run-swop scripts/bin/run-swop
index 1987e15..e8a30ba 100755
--- scripts/bin/run-swop
+++ scripts/bin/run-swop
@@ -97,6 +97,8 @@ processComponentBatchOpt()
         --swop-uid) checkAndAssign "${opt}" "${optarg}"
             re='^[0-9]+$'
             [[ ! "${optarg}" =~ ${re} ]] && error "${opt}: ${optarg}: bad user id";;
+        --cgroupns) checkAndAssign "${opt}" "${optarg}"
+            ;;
         *) unprocessedOpts+=("${origParam}"); nShift=1;;
     esac

@@ -126,6 +128,8 @@ onTrainEnd()
     envvar+=(-e "SWOP_GID=${dockerGroupId}")
     envvar+=(-e "SWOP_PROFILE=${profileFileName}")

+    [[ -n "${cgroupns}" ]] && miscDockerRunParams+=(--cgroupns ${cgroupns})
+
     cmd+=("${unprocessedOpts[@]}")
     unprocessedOpts=()

And, I added to the command line option of the script and it works on Ubuntu 22.04.--cgroupns host``run-swop

@IMOKURI Thank you for your help. It is indeed a problem with the Ubuntu version. Your changed script works well on Ubuntu 22.04.

I have the same issue, the container id cannot be extracted.
For me the above mentioned changes are not working. I still get the same error.
On my CentOS 8 system, both version of cgroups are supported.
$ ~/git/swarm-learning > grep cgroup /proc/filesystems
nodev	cgroup
nodev	cgroup2

Any ideas?

Since CentOS8 uses cgroup v1 by default, I don't think this workaround is necessary.
If you still get the Unable to extract container ID error, there may be another problem.
It is better to submit a new issue according to the issue template.

In the released https://github.com/HewlettPackard/swarm-learning/releases/tag/v1.1.0, this workaround https://github.com/HewlettPackard/swarm-learning/issues/103#issuecomment-1164030572 is not required.

Closing this issue, as it is addressed in latest 1.1.0 release.


.localplaylst extensions fix

I was wondering why bot tries to play audiochecker.log files...

Any update on this @FredericaBernkastel ?


Allow automatic detection of solvated PDBs in PointMutationExecutor

Upon discussing #1024, we decided that we would like to write a code snippet that automatically checks whether we should solvate the system or not (perhaps inside the _solvate() method) such that it warns (or errors out) if users are trying to solvate a system that already has been solvated.

Note that this may be tricky, as if we are doing a protein mutation in a protein-small molecule system, the small molecule will be specified as an SDF, which I'm not sure can contain waters

I should also note that some PDBs may contain crystal waters, but do not actually have a solvent box, so we'll want to make sure we aren't just checking for the presence of one water, as this will make the PointMutationExecutor incorrectly assume that the PDB does not need to be solvated.


Be quiet on success

Unix commands should not output anything on success.  This causes lots of extraneous output during npm install under a certain common misconfiguration involving $PATH (#8, brianloveswords/nvm#1).

Fixed on v2


Multiple conflicting sets of iptables rules

When playing with the neutron-calico Juju charm I found my VMs were unable to communicate with each other. After further investigation I found that the problem was that nova-network, which had previously been running on the box, had not removed its iptables rules when it was terminated.
Because of the OpenStack utilities for combining iptables rules, these rules didn't cause conflicts with calico-compute, so they went undetected. However, they were not updated to reflect the new state of the network. This meant that new machines had their traffic blocked by the nova-network rules, even when the calico-compute rules would have allowed it through.
I resolved this temporarily by flushing the rules on my system and then restarting calico-compute (to repopulate its rules). This works in the simple case but may not work generally. We need a way to spot whether calico-compute is the only entity managing iptables rules.

This appears to be a duplicate of issue #80, at least partially.


Rename test suite due to feedback and add EE and MP test cases

Rename scanner-it to binary-scanner-it and add tests for EE6 without MP, EE7 with MP v1, EE8 with MP v2 and v3, Jakarta EE8 with MPv4 and Jakarta EE9.1 with MP v5. These are the compatible versions which are supported, we do not try to test incompatible versions.
Some of the code is repetitive and it is possible to crunch it down smaller but that would make it more complex and more difficult to support. So this is supposed to be a balance.

Looks like scanner-it is running on the GH actions - is this expected?

Thanks for pointing out the scanner-it run. I needed to check in the deletion of the files resulting from the renaming.


Incorrect note fetched

I have periodic notes configured with the format YYYY/MM/DD, which creates a file day.md in the year/month directory.  Journal review doesn't appear to load the proper notes. For example, if it's 2024/06/08, it will load 2024/04/07 as "1 day ago". It seems that the year and day are considered during the matching, but for some reason the month isn't, which results in notes from April showing up as one day ago in June.
I also have Journal review display notes from "every 1 week" and it's not returning any matches when it should. Let me know if I should file this as a separate issue.
Periodic notes: 1.0.0-beta.3
Journal review: 2.3.2

Hey, thanks for the message.
Can you send me a screenshot of your folder structure (without any note contents, of course)? Just to confirm how it looks?
And for the "display notes from every 1 week", do you mean weekly notes? Because these are not supported yet. Or do you just mean daily notes from 1, 2, 3, etc. weeks ago?
I haven't tested the plugin with Periodic notes: 1.0.0-beta.3., but will give that a try too.

As requested, a screenshot of the file structure.

As for displaying notes from every 1 week, I want to see the daily notes, not the weekly notes.


Alternative to sort-version?

I'm building my package inside a manylinux1 container (CentOS 5), so that the dylibs can be used in Python wheels. This used to work perfectly, but the change to cross and its dependence on sort --sort-version is difficult to fix; I can't even upgrade to the latest coreutils, because CentOS 5 doesn't support the .xz format that's used after coreutils version 8.13
I know gsort can be installed using homebrew on MacOS, but I wonder if there's a simple way to fix this…

(Fixed by installing from source)


Improve abs url replacement speed

This commit replaces the multuple bytes.Containts and bytes.Replace with with a custom replacer that does one pass through the document and exploits the fact that there are two common prefixes we search for, src= and href=.
This is both faster and consumes less memory. There may be even better algos to use here, but we must leave some room for improvements for future versions.
This should also make it possible to solve #816.
benchmark              old ns/op     new ns/op     delta
BenchmarkAbsUrl        25795         22597         -12.40%
BenchmarkXmlAbsUrl     17187         11166         -35.03%

benchmark              old allocs     new allocs     delta
BenchmarkAbsUrl        60             33             -45.00%
BenchmarkXmlAbsUrl     30             16             -46.67%

benchmark              old bytes     new bytes     delta
BenchmarkAbsUrl        5844          4167          -28.70%
BenchmarkXmlAbsUrl     3754          2069          -44.89%

Fixes #894

Great stuff. Merged as d33a42f


Unable to exclude a field

I might just be missing this in the code but I'm having some difficulty figuring out how to exclude subdocuments from a result.  One of the collections I'm working with can have fairly extensive array of subdocuments and the unserialize() functionality takes a fair amount of resources.  I've moved the functionality into an aggregate() function (which is awesome BTW) but I would like to omit retrieving those subdocuments from my normal queries so I don't have to bear with the overhead of unserialization.
In the old MongoDB Collection I could specify
db.inventory.find(
   { type: 'food', _id: 3 },
   { "classification.category": 0}
)

but I can't find how that can be accomplished with this library.

The Collection::find() and Collection::findOne() methods take a projection option, which corresponds to the second argument in your example. This is equivalent to the projection option on the MongoDB\Driver\Query object in the underlying driver.
This will be properly documented in https://github.com/mongodb/mongo-php-library/pull/144. To copy/paste an example from that PR:
$document = $collection->findOne(
    ['_id' => '10011'],
    ['projection' => ['pop => 1']]
);

And there's 5 seconds off my elapsed execution time.  Thanks a bunch.


Error when training custom object with tensorflow.

###System Information
Have I written custom code(No)
OS Platform and Distribution(Linux 16.04 LTS)
TensorFlow installed from(python3 package)
TensorFlow version(1.5.0)
CPU(Intel Core i5-3320M)
I am trying to train my custom object by using tensorflow, however, when I run command it shows an error with protobuf:
File "/usr/local/lib/python3.5/dist-packages/google/protobuf/text_format.py", line 1152, in ConsumeIdentifierOrNumber raise self.ParseError('Expected identifier or number, got %s.' % result)
google.protobuf.text_format.ParseError: 235:1 : Expected identifier or number, got <.
I am using following command to run train.py:
python3 train.py --logtostderr --train_dir=training/--pipeline_config_path=training/ssd_mobilenet_v1_pets.config
Also I am attaching screenshot with the error.

I am beginner in this topic, so I would be grateful for any help.

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

Nagging Awaiting Response: It has been 14 days with no activityand the awaiting response label was assigned. Is this still an issue?

iI met the same problem,have you solved it?

I face with the same issue. Any idea how to solve it?

I faced the same problem, to solve it I had to take the format of mscoco_label_map.pbtxt which is different
item {
name: "/m/01g317"
id: 1
display_name: "person"
}

you can file the above mentioned file "mscoco_lab_map.txt in models/research/object_detection/data


i added  a new category of commands to the outline titled 'Linux Time…

…zones commands'

Thank you for creating this project, it is an insightful one. Hopefully i can contribute more commands in the future as i keep learning.


WIP

I took a shot at updating and refactoring this: https://github.com/sargunv/async-task-graph/pull/3/


Support symlinks with VFS retention

cc: @gradle/build-cache

This would mainly mean that we'd need to canonicalize paths all the time, something we stopped doing as a performance improvement a while back.

I think it is a little bit more complicated than that. When a directory contains a symlink, then it isn't enough to watch the directory for changes but we also need to watch the parent directory of the target of the symlink.
Currently, we also only normalize the roots of whatever we are snapshotting, so canonicalizing those also wouldn't be enough. I think as a start it would be better to detect the symlinks and drop the VFS entries for the symlinks at the end of the build.

As a first step, we could detect when we try to watch directories which have a different canonical path than the requested path and then disable VFS retention.
For a hierarchical watcher, we need to check that all the subdirectories we care about (aka which we want to watch), do not contain symlinks. For a non-hierarchical watcher we need to make sure that all the directories we pass to the watcher are not symlinks. We probably also need to check that no files in the watched directories are symlinks.
The detection can happen while we snapshot a directory or a file, since there we already know if a directory/file is a symlink.

Fixed via https://github.com/gradle/gradle/pull/13031.


[url_launcher] Update readme for iOS targeting iOS 9.0 and higher

Description
Add instructions for iOS, for targeting iOS 9.0 and higher.
Note:
All new projects 1.22+ targets iOS 9.0 by default

Related Issues
Related to https://github.com/flutter/flutter/issues/65936
Checklist
Before you create this PR confirm that it meets all requirements listed below by checking the relevant checkboxes ([x]). This will ensure a smooth and quick review process.

[x] I read the Contributor Guide and followed the process outlined there for submitting PRs.
[x] My PR includes unit or integration tests for all changed/updated/fixed behaviors (See Contributor Guide).
[x] All existing and new tests are passing.
[x] I updated/added relevant documentation (doc comments with ///).
[x] The analyzer (flutter analyze) does not report any problems on my PR.
[x] I read and followed the Flutter Style Guide.
[x] The title of the PR starts with the name of the plugin surrounded by square brackets, e.g. [shared_preferences]
[x] I updated pubspec.yaml with an appropriate new version according to the pub versioning philosophy.
[x] I updated CHANGELOG.md to add a description of the change.
[x] I signed the CLA.
[x] I am willing to follow-up on review comments in a timely manner.

Breaking Change
Does your PR require plugin users to manually update their apps to accommodate your change?

[ ] Yes, this is a breaking change (please indicate a breaking change in CHANGELOG.md and increment major revision).
[x] No, this is not a breaking change.

CC: @cyanglaz

@hamdikahloun
Is there anything we need to merge this?
Thank you

hi @TahaTesser
Last time i checked it, the branch has conflicts that must be resolved .
Please fix the conflicts before merging .
Thank you

@hamdikahloun
Just rebased and bumped version, update changelog
I've removed the When targeting iOS 9.0+, new default from Flutter 1.22+. line since this version of stable has been around for months
Can you please review that current updated PR?
Thank you

@hamdikahloun
Just rebased and bumped version, update changelog
I've removed the When targeting iOS 9.0+, new default from Flutter 1.22+. line since this version of stable has been around for months
Can you please review that current updated PR?
Thank you

LGTM

LGTM

LGTM

@stuartmorgan
Sorry for the delay, made the changes.

I've restarted the failed test, since it was a network issue.


Add support for pipeline parameters

Is your feature request related to a problem? Please describe.
Pipeline parameters enable users to provide runtime parameters when submitting or exporting a Kubeflow Pipelines pipeline via the Pipeline Editor or the CLI, without the need to modify the .pipeline file. Pipeline parameters can also be provided when running a pipeline from the Kubeflow Pipelines user interfaces, such as the Central Dashboard.
Describe the solution you'd like

Pipeline editor:  allow for the definition of pipeline parameters
Pipeline editor: allow for the assignment of pipeline parameters to custom component properties
Pipeline editor: allow for the assignment of pipeline parameters to generic component properties
Pipeline editor: extend the export wizard to allow for specification of pipeline parameters
Pipeline editor: extend the submit wizard to allow for specification of pipeline parameters
elyra-pipeline CLI: update describe command to list pipeline parameters that can be specified when exported

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context

Requires https://github.com/elyra-ai/elyra/pull/2993
Issues that raised this requirement:

https://github.com/elyra-ai/elyra/issues/2858
https://github.com/elyra-ai/elyra/issues/1903
https://github.com/elyra-ai/elyra/discussions/2985
(please add)

Delivered in https://github.com/elyra-ai/elyra/pull/3053 and https://github.com/elyra-ai/elyra/pull/3001


Small typo in map dragging option description

OK done.
By the way, that was my first time using a "Codespace" to fix up a PR and it was amazing;

It launched a VSCode-like instance in a browser tab. I edited from there and committed using the "VS Code" terminal


Super Slow under Load with file responce

First Check

[X] I added a very descriptive title to this issue.
[X] I used the GitHub search to find a similar issue and didn't find it.
[X] I searched the FastAPI documentation, with the integrated search.
[X] I already searched in Google "How to X in FastAPI" and didn't find any information.
[X] I already read and followed all the tutorial in the docs and didn't find an answer.
[X] I already checked if it is not related to FastAPI but to Pydantic.
[X] I already checked if it is not related to FastAPI but to Swagger UI.
[X] I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

[X] I commit to help with one of those options 👆

Example Code
@app.get("/download/{channel_id}/{base_url:path}", response_class=FileResponse)
async def download_segments(channel_id: str, base_url: str):

    URL_PHARSE  = urlparse(base_url)
    FILE_NAME   = os.path.basename(URL_PHARSE.path)

    if "/audio/" in base_url:
        SEGMENT_TYPE = "audio"
    else:
        SEGMENT_TYPE = "video"

    SEGMENT_ID      = FILE_NAME.replace(".m4s","")
    CONTENT_TYPE    = SEGMENT_TYPE + "/mp4"

    FILE_LOCATION = await GET_CACHE_PATH("segments" ,True, channel_id) + SEGMENT_ID + "_" + SEGMENT_TYPE + ".m4s"

  return FileResponse(FILE_LOCATION, media_type=CONTENT_TYPE)

Description
I am trying to create a dash streaming server using fast api and uvicorn everything works absolutly perfect and a custom dash manifest is built from my main endpoint and then the second endpoint in the example grabs and provides the segments back to the client
my issue comes when there are more than 4 clients trying to view media as soon as 4 or more people try connecting then the streams start to stutter and pause as soon as it drops back down to 3 users or below the streams work perfect again
the way thesystem works is i have all the segments in ramdisk storage and when a user tries to play it in a media player the download section of the code is called and the file is provided back to the user to me it seems that once 4 or more people connect the operation of returning the files slows down drastically
Operating System
Linux
Operating System Details
0S : UBUNTU 18.04
RAM : 128GB
STORAGE : 500GB SSD
BANDWIDTH : 1GBPS DEDICATED
FastAPI Version
Latest
Python Version
3.10
Additional Context
i have the same kind of system up and running on another server using nginx and php and it works good no issues even with 30/40 people connected but the upkeep and amount of code needed is like 100x that of python that is why i was looking at switching to fast api

I can't think of anything immediately obvious but here are some things I would check, in no particular order
Look at the file response object and see if it is behaving in a way you do not expect.
https://github.com/encode/starlette/blob/master/starlette/responses.py#L279
If you dont notice anything in the above try opening the file yourself and  use just a regular response object
You should probably check to see you are not saturating your disk access in some way with something like iotop or exhausting your ram and causing things to page to disk with something like top/htop
Also how are you deploying this? How many workers are you running? If you are just running the single python process is working hard or not ?

I can't think of anything immediately obvious but here are some things I would check, in no particular order
Look at the file response object and see if it is behaving in a way you do not expect. https://github.com/encode/starlette/blob/master/starlette/responses.py#L279
If you dont notice anything in the above try opening the file yourself and use just a regular response object
You should probably check to see you are not saturating your disk access in some way with something like iotop or exhausting your ram and causing things to page to disk with something like top/htop
Also how are you deploying this? How many workers are you running? If you are just running the single python process is working hard or not ?

I am running this locally now using uvicorn in the python code and starting it with python3 main.py
i have checked htop and the system uses little to no power at all and all files used by the python code are stored in a tempfs ramdisk of 40gb to avoid killing the storage and hitting any storage io issues

Probably I would try with just a regular response, and set everything manually and see if that side steps the issue.
Also make sure you don't have any blocking code anywhere that might be blocking the event loop in some way.

Probably I would try with just a regular response, and set everything manually and see if that side steps the issue.
Also make sure you don't have any blocking code anywhere that might be blocking the event loop in some way.

i will give it a try with a normal responce now as for blocking code i dont think there is any as everything runs fine for 1-3 open streams anything over is what kills it thats why i was thinking it had to be something todo with the responce times that got worse as more was opened

Well if you have blocking calls happening that effectively makes the event loop pause and wait for those calls to complete and it will not make any progress on other tasks/requests while it is blocked.  More requests would equal more blocking which equals more pausing.
Is you program doing any non-async file or network io or using a library that might spend a long time between calls to functions with await

Have you tried using gunicorn instead of uvicorn? Launching more workers might help when you have multiple clients access your method.


Provide null or undefined children

I have hit this bug again, it's pretty much everywhere we need a global function to clear null or undefined children props instead of using this.props.children e.g. this is better this.props.children.filter(child => child)
CardItem.js:114
<Card>
  <CardItem>
    .......
   {isLoading && <Loading />} // error here cant work with null or undefined
  </CardItem>
</Card>

This bug happens all around the library.

FooterTabhas the same issue, when adding a Button with a conditional Badge.
        <FooterTab>
           <Button {...bprops}>
              {badges > 0 ? (
                <Badge>{badges}</Badge>
              ) : null}

It is not filtering the falsy children:
https://github.com/GeekyAnts/NativeBase/blob/master/Components/Widgets/FooterTab.js#L76-L89

Hi, we acknowledge it's a major bug and we're working on it. Thanks for pointing it out!

Fixed with 0.5.20


Create pl_pl.json

Polish translation to this mod

Hi, thanks a lot for your input! I am expanding on localization on the 0.1.0 branch if you'd care to take a look:
https://github.com/TerraForged/TerraForged/blob/0.1.x/generated/resources/assets/terraforged/lang/en_us.json


build error with new serenity project

I created a new poject (asp.net, the version of Serenity is <IP_ADDRESS>), when i try to build the project,i always got below error.
This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is ....\packages\Microsoft.TypeScript.MSBuild.3.4.0\build\Microsoft.TypeScript.MSBuild.props. NIKI.Web	D:\02_工作\05_我的项目\NIKI\NIKI\NIKI.Web

Hi @Nathan2020-0126 ,
So did you try what the error suggests? To perform a nuget Package restore?
See: https://docs.microsoft.com/en-us/nuget/consume-packages/package-restore
The problem is that on your computer, not all necessary nuget packages to run the solution are installed.
With kind regards,
John

@JohnRanger , I have try retore, but i didnot work.
After I deleted below lines in csproj file, then I can successfully build project. But I don't konw if it would cause other problems.
 <Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
    <PropertyGroup>
      <ErrorText>This project references NuGet package(s) that are missing on this computer. Use NuGet Package Restore to download them.  For more information, see http://go.microsoft.com/fwlink/?LinkID=322105. The missing file is {0}.</ErrorText>
    </PropertyGroup>
    <Error Condition="!Exists('..\..\packages\Microsoft.TypeScript.MSBuild.3.4.0\build\Microsoft.TypeScript.MSBuild.props')" Text="$([System.String]::Format('$(ErrorText)', '..\..\packages\Microsoft.TypeScript.MSBuild.3.4.0\build\Microsoft.TypeScript.MSBuild.props'))" />
    <Error Condition="!Exists('..\..\packages\Microsoft.TypeScript.MSBuild.3.4.0\build\Microsoft.TypeScript.MSBuild.targets')" Text="$([System.String]::Format('$(ErrorText)', '..\..\packages\Microsoft.TypeScript.MSBuild.3.4.0\build\Microsoft.TypeScript.MSBuild.targets'))" />
  </Target>

There is an error about transfer T4 template.
Below is C# class

Below is the typescript

There is an error about cannot find namespace serenity. I  don't know if this error is related with the action that I delete some lines in csproj file.

Hi @Nathan2020-0126
your first statement where you say that it completes after you deleted certain error conditions which check if you have MS typescript installed, makes me think that you indead do not have typescript correctly installed. Please make sure that - under extensions in Visual Studio - you install a recent typescript version:

And please put back the error condition checks. Removing these is like taking strong pain killers when you have serious pain - instead of curing the problem :-)
Your project has to pass them - if it does not, your project (or the current setup of your computer) does indeed not meet the requirements which SF needs to successfully compile.
Some of the most important conditions are:

current NodeJS installed

recent TypeScript version installed

If your computer is behind a corporate firewall and you must use a proxy to access the internet, this often causes problems. If this is the case, try to find a way to connect your computer directly to the internet - if that is allowed by your corporate IT. If not, then you have to google for the following themes:

NPM behind proxy

nuget package manager behind proxy

visual studio behind proxy

Please check the above.
With kind regards,
John

I installled TypeScript 3.8.2 for Visual Stuido, but it still didnot work.
I just created a  .Net Core version project,  I can successfully build it.

you might need to install nodejs

check more here https://serenity.is/docs/getting_started/README

Hmm - that's strange. I just very recently have created a new (real) .NET  SF project from the visual studio template (3.9.14) and that worked without any problem.
But I have quite some problems with new .NET core projects as - at least when I do it - the generated Serene projects do lack a lot of what goes within /wwwroot/.
(Copying them over from a real .NET SF project does solve this problem though.)

@minhhungit I have installed node js, still didnot work.
@JohnRanger @minhhungit  thank for you both very much.


Why is a init required?

After upgrading snag i noticed a configuration file is now required in the directory I want to use it in. The readme mentions sane default ignores and the created .config.yml is an example of things I can customize so I am trying to understand why the init per directory is now required? :neckbeard:

The purpose of the init command was to allow users to create a quick .snag.yml file in whatever directory they're in. While running the command is not required, having a .snag.yml file in the directory that you're running snag in has always been required. The only way to bypass having that file in your directory is to use the -c flags when running the snag command (Refer to the quick-use section of the README).
If you'd like to give us some more details as to your project structure and how you're using snag, we'd be happy to help find a solution that works for you.

Thank's for the quick response! The version of snag I was using does not seem to actually have a -version option :D but it is/was running go build/test/vet within ./... "package path" on each cycle so after finally upgrading to 1.+ I was expecting the same golang biased defaults. Defiantly like the language/neutral tool direction and combination of using -c with list of commands and/or config 👍

Glad you were able to resolve your issue! Please feel free to reach out again if you come across any issues or have a feature request. Happy snagging. 🍰


Removing unuseful/unused commands from daemon API

Remove unuseful/unused commands from Daemon API
get_nametrie
log
upload_log
get_mean_availability

          1 Error

:no_entry_sign:
Please include a CHANGELOG entry.

          1 Message

:book:
See http://keepachangelog.com/en/0.3.0/ for details on good changelog guidelines

  Generated by :no_entry_sign: Danger

also delete open and reveal commands, and their respective functions in file_utils.py


HDDS-8209. [SNAPSHOT] Synchronize tarball creation with background processes.

What changes were proposed in this pull request?
The BootstrapStateHandler
This PR creates a new interface, BootstrapStateHandler.  Each process that manages state that needs to be copied into the bootstrap tarball must implement this interface.
The interface has a single method, getBoostrapStateLock().  The processes managing bootstrap state implement this method, which returns the lock used to protect state that must be changed atomically with respect to the tarball creation process.
The tarball creation process takes each of these locks before generating the tarball.  That prevents any of the processes from changing the state while the tarball is being created.
Then the tarball creator waits for the double buffer to flush so that any remaining operations that may effect the bootstrap state have completed before the tarball is created.
The state to protect/synchronize
Outside of the active rocksdb itself, the omSnapshot subsystem uses many types of persistent state data.  I've listed the types below along with an indication of whether they are guarded by a BootstrapStateHandler.Lock.
If there is any type I've forgotten please let me know, so that we can account for it.
delete/rename key entries
The sdt, and eventually the kdt, move delete/rename key entries between snapshots.  These are guarded by a BootstrapStateHandler.Lock, (and by waiting for the double buffer flush.)
deleted sst files
The rocksdb differ and sst filtering service delete sst files when no longer needed.  These are guarded with by the BootstrapStateHandler.Lock.
The reason for these need to be guarded is that the tarball creation process does a calculation of hard links prior to tarball creation.  That calculation requires a stable set of sst files.  If some get deleted during the process, the hard links calculated may be invalid.
compaction logs
These get created by the active rocksdb as sst files are compacted.
These are not currently guarded by a BootstrapStateHandler.Lock.
My reasoning for this is that it would require turning off compactions on the active rocksdb for the duration of the tarball creation.  Before the tarball is created, the BootstrapStateHandler's are locked and a checkpoint of the active fs is taken.  Any compactions that happen after that checkpoint is taken, but before the tarball is finished, will have compaction log entries add to the tarball, that don't correspond to the checkpoint.
This may be a problem.  If so, we'll have to pause compactions on the active fs long enough to make a copy of the compaction logs.  So please include your thoughts when reviewing.
intermediate rocksdb snapdiff files
These are just used during the calculation of the snapdiff.  They will not be a part of the tarball.
They are not guarded by a BootstrapStateHandler.Lock.
SST filter service history file
This file keeps track of all the snapshots that have been filtered.
It is guarded by a BootstrapStateHandler.Lock.
Other changes in this PR
Flush Snapshot WAL's after moving deleted keys
When deleted keys are moved from one snapshot to another, the double buffer opens and writes to both the corresponding rocksdb images, in addition  to the rocksdb for the active fs.  I've modified that operation to flush the wal for the snapshot rocksdb images.  (The active rocksdb doesn't need to be flushed because the tarball creator takes a rocksdb checkpoint of that image after the doublebuffer is flushed.)
Moved some methods to the AbstractKeyDeletingService class
The AbstractKeyDeletingService class is the parent class of both the SnapshotDeletingService and the KeyDeletingService.  I moved the submitSnapshotMoveDeletedKeys() and submitRequest() methods from the SnapshotDeletingService class to the AbstractKeyDeletingService class.  This is because thesnapshot delete design includes an optimization that has the KeyDeletingService also moving deleted keys.
Since that operation needs to be protected by a BootstrapStateHandler.Lock, I moved that code in anticipation of when the optimization is implemented.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-8209
How was this patch tested?
added unit tests
I will add a test for the SnapshotDeletingService once this is merged:  https://github.com/apache/ozone/pull/4571
TODO
As mentioned above, we need to decide if the compaction logs need to be synchronized with the actvie fs checkpoint.

@hemantk-12 @prashantpogde @aswinshakil @smengcl   Please take a look.

do you think it's reasonable to call pauseBackgroundWork() at the beginning of bootstrapping, so that all RDB background work, including compaction, would be paused?

I don't know much about compaction so I'm not sure.  It depends on how long compactions last, and how bad is it to turn them off for a while.  Will that cause problems for the active fs?  My gut feeling is that it is best to add it so we can do more testing and see if the side effects are acceptable.

it could still append new compaction log entries (and hardlink new SST files) in CompactionBegin/CompletedListener.

I don't think the new sst files are a serious problem.  The question is are the new compaction log entries a problem.  That is what I need help determining.  If they are not a problem, then we don't need to pause compactions.
The issue is after the bootstrap process takes the checkpoint of the active fs, any compactions before the tarball is finished will create entries to the comaction logs.  Those entries don't correspond the active fs in the tarball.  Once the follower loads the tarball, it will start doing it's own compactions which may differ from the compaction log entries.  It could cause conflicting entries in the compaction log, which might break the dag.  That is the problem we may need to avoid by invoking pauseBackgroundWork().
For example, the leader could compact files a and b into file c while the follower could compact them into file d.  What is the compaction dag supposed to do with conflicting entries like that?  That is why I'm leaning towards stopping compactions.

They do call locks internally. Do you think that is insufficient?

Ah that is good enough. Thanks. They count as rocksDbCheckpointDiffer in the Lock I presume:
https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OMDBCheckpointServlet.java#L304-L309

I don't think the new sst files are a serious problem. The question is are the new compaction log entries a problem. That is what I need help determining. If they are not a problem, then we don't need to pause compactions.

I agree with sufficient handling in DAG traversal (e.g. when used in SnapDiff) this should not be a big problem. cc @hemantk-12
I think alternative approach to pauseBackgroundWork() that could also keep compaction log consistent could be that we ask bootstrapping followers to ignore any newer compaction log entries than the checkpointed active DB's sequence number, because any compactions happened after that sequence number, even persisted to compaction log right during tarball creation, would be irrelevant to the active DB checkpoint. What do you think?

we ask bootstrapping followers to ignore any newer compaction log entries than the checkpointed active DB's sequence number,

I think that is a great solution if it is not too difficult.  Does each compaction log entries have a corresponding sequence?  How do we new which entries are newer than a certain number?

we ask bootstrapping followers to ignore any newer compaction log entries than the checkpointed active DB's sequence number,

I think that is a great solution if it is not too difficult. Does each compaction log entry have a corresponding sequence number? How do we new which entries are newer than a certain number?

Here is an example of compaction log that is used in a UT:
https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java#L281-L302
So it looks like atm we only have sequence number for S  (snapshot taken) event entries. For a cleaner solution we should prepend sequence number to those C  (compaction) entries as well. (At the time I thought about adding sequence number for C  entries but didn't because we didn't have a use for it back then. Now is the time to add it.)
So by that I mean this line:
https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/test/java/org/apache/ozone/rocksdiff/TestRocksDBCheckpointDiffer.java#L285
should become something like:
"C 2004 000001,000002:000062\n"

This could be achieved by adding this line:
sb.append(db.getLatestSequenceNumber()).append(SPACE_DELIMITER);

right after this:
https://github.com/apache/ozone/blob/08cb5208b95abbccfada0d16bbe65a0451d8a88b/hadoop-hdds/rocksdb-checkpoint-differ/src/main/java/org/apache/ozone/rocksdiff/RocksDBCheckpointDiffer.java#L545-L546
And we will also need to tune the compaction log reading/parsing to correctly handle this during OM startup.
You could leave a TODO in this PR for this, and file a new jira for the compaction entry sequence number addition. (up to you)

Thank you @GeorgeJahad for this PR. Also thank you for the detailed abstract. Yes, we should call pauseBackgroundWork() during tarball creation and be sure to resume it once the tarball is done. Other changes look good to me.

You could leave a TODO in this PR for this, and file a new jira for the compaction entry sequence number addition. (up to you)

That would be my preference.  I think it is a much better solution than the pausing compactions.  Is that ok with you @prashantpogde ?

@smengcl I created a new ticket for the compaction log work: https://issues.apache.org/jira/browse/HDDS-8652

Thanks @GeorgeJahad for the patch and filing HDDS-8652.
Thanks @prashantpogde for the review.

@GeorgeJahad @smengcl

You could leave a TODO in this PR for this, and file a new jira for the compaction entry sequence number addition. (up to you)

That would be my preference. I think it is a much better solution than the pausing compactions. Is that ok with you @prashantpogde ?

It's a better solution but Its much simpler to just pause and resume the background compaction. I would favor simplicity over the other solution because sync up with leader would be an infrequent activity. I would leave it to you to decide on this.

@GeorgeJahad @smengcl

You could leave a TODO in this PR for this, and file a new jira for the compaction entry sequence number addition. (up to you)

That would be my preference. I think it is a much better solution than the pausing compactions. Is that ok with you @prashantpogde ?

It's a better solution but Its much simpler to just pause and resume the background compaction. I would favor simplicity over the other solution because sync up with leader would be an infrequent activity. I would leave it to you to decide on this.

Another downside of pauseBackgroundWork() is that it waits for ALL RocksDB background work to finish before pausing them, not just compaction. That includes any ongoing memtable flush as well, which is unnecessary in our use case here. At this moment I don't see RocksDB implementing API to pause only compaction. Thus, IMO pauseBackgroundWork() is not ideal and I would only consider it when we don't have other choices (which we do now).


Node version requirements

Does this require node version greater than or equal to 18? I received the following error when I tried to do a yarn add spin-delay:
error<EMAIL_ADDRESS>The engine "node" is incompatible with this module. Expected version ">=18.0.0". Got "16.20.2"
error Found incompatible module.

"error @azure/logger@1.1.2:"
Wrong repo? We only have a peer dependency on react, so I don't think that error is ours.


Make RSpec/BeNil cop configurable

Make RSpec/BeNil cop configurable with a :be_nil and a :be style.
Inspired by @marcandre's comment at https://github.com/rubocop/rubocop-rspec/pull/1239#issuecomment-1067532133

I see absolutely no reason to prefer be_nil to be nil. The latter is stricter, simpler and requires less knowledge of rspec.

Before submitting the PR make sure the following are checked:

[x] Feature branch is up-to-date with master (if not - rebase it).
[x] Squashed related commits together.
[x] Added tests.
[x] Updated documentation.
[x] Added an entry to the CHANGELOG.md if the new code introduces user-observable changes.
[x] The build (bundle exec rake) passes (be sure to run this locally, since it may produce updated documentation that you will need to commit).

If you have modified an existing cop's configuration options:

[x] Set VersionChanged in config/default.yml to the next major version.

Let’s get #1246 merged first, before merging this PR.


Implement variable argument callbacks

This commit adds support for a variable-argument callback, instead of just the 0, 1, or 2 supported by the current code.  In addition to this commit, h$makeCallbackApply needs to have the following addition
  } else if (n == -1) {
      c = function() {
          var callbackArgs = new Array(arguments.length);
          for (int i = 0; i < arguments.length; i++) {
              callbackArgs[i] = arguments[i];
          }
          var args = extraArgs.slice(0);
          var action = (h$c2(h$ap1_e,(fun),((h$c1(h$ghcjszmprimZCGHCJSziPrimziJSRef_con_e, (callbackArgs))))));
          args.unshift(action);
          return f.apply(this, args);
      }

which I can send in a separate pull request if you like this approach.

Also, if you don't like this approach (it is kinda odd), I can submit a pull request with just a
syncCallbackN :: OnBlocked -> (JSArray -> IO ()) -> IO (Callback (JSArray -> IO ()))
syncCallbackN onBlocked x = js_syncCallbackApply (onBlocked == ContinueAsync) (-1) (unsafeCoerce x)

with the same change to h$makeCallbackApply.

Sorry, this got closed due to cleaning up branches. Please reopen against master.