Fix YouTube's autogenerated subtitles

Checklist

[X] I'm reporting a feature request
[X] I've verified that I'm running yt-dlp version 2<IP_ADDRESS>. (update instructions)
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[x] I've read the guidelines for opening an issue

Description
When you download automatic subtitles from YouTube, the resulting subtitle is a rolling subtitle - every time a new line is added, the previous one is moved up a line - if there's more than two lines, the first one disappears. Think Star Wars intro, but with only two lines:
A subtitle converted from VTT to SRT by yt-dlp would look something like this
00:00 --> 00:03
This is the first line

00:03 --> 00:10
This is the first line
This is what happens when another line is added

00:10
This is what happens when another line is added
If a third one is added, the first one disappears and the second one shoots up

The problem with this is that it's really hard to read, since you expect both lines to change, and it becomes really really (reeeally) distracting. I needed subtitles from my classes to be searchable and easily read while playing, so I made a little script to fix the VTT's and manually create a new SRT, but it becomes a boring step every time I need to download a new class to download, convert and embed. Is the community interested in something like this to automatically fix Youtube's autogenerated subtitles?
My ~sketchy~ fix is as follows
def fix_youtube_vtt(vtt_file_path) -> str:
    """Fixes Youtube's autogenerated VTT subtitles and returns a srt-formatted string"""

    import webvtt

    pretty_subtitle = ''
    previous_caption_text = ''
    i = 1
    for caption in webvtt.read(vtt_file_path):

        if previous_caption_text == caption.text.strip():
            # if previous and current lines are `identical`, print the start time from the previous
            # and the end time from the current.
            pretty_subtitle += f"{i}\n{previous_caption_start} --> {caption.end}\n{previous_caption_text}\n\n"
            i += 1

        elif previous_caption_text == caption.text.strip().split("\n")[0]:
            # if the current caption is multiline, and the previous caption is equal to
            # the current's first line, just ignore the first line and move on with the second.
            previous_caption_text = caption.text.strip().split("\n")[1]
            previous_caption_start = caption.start
            last_caption_end = caption.end

        else:
            previous_caption_text = caption.text.strip()
            previous_caption_start = caption.start.strip()

    return pretty_subtitle

The output becomes:
00:00 --> 00:03
This is the first line

00:03 --> 00:10
This is what happens when another line is added

00:10
If a third one is added, the first one disappears and the second one shoots up

In no way this is an ideal solution, I'm just showing what I've done to circumvent the issue, but a few flaws in my solution are that it only outputs a single line, and the timing is slightly off (maybe in the double digit milliseconds) due to the original multi line  captions.

I don't think this makes sense in the core program. But you can convert this into a plugin postprocessor. See https://github.com/yt-dlp/yt-dlp#plugins and example

That said, if a fully native implementation of this can be made (without using the webtt-py library), I would not be opposed to using that instead of the current ffmpeg-based subtitle conversion.
Also see https://github.com/yt-dlp/yt-dlp/issues/1625

so has this been fixed in the branch or is there still some weird workaround?

I created an plugin to fix the subtitles. Hope it's useful to someone. There is also a standalone exe with GUI available.

changes can be made according to language
yt-dlp --embed-subs --merge-output-format mkv -f 'bv+ba' --write-auto-subs --sub-langs 'en' 'https://youtu.be/3_HG33-IYaY' --sub-format ttml --convert-subs srt --exec 'before_dl:fn=$(echo %(_filename)s| sed "s/%(ext)s/en.srt/g") && ffmpeg -fix_sub_duration -i "$fn" "$fn".tmp.srt && mv "$fn".tmp.srt "$fn"'

changes can be made according to language yt-dlp --embed-subs --merge-output-format mkv -f 'bv+ba' --write-auto-subs --sub-langs 'en' 'https://youtu.be/3_HG33-IYaY' --sub-format ttml --convert-subs srt --exec 'before_dl:fn=$(echo %(_filename)s| sed "s/%(ext)s/en.srt/g") && ffmpeg -fix_sub_duration -i "$fn" -c:s text "$fn".tmp.srt && mv "$fn".tmp.srt "$fn"'

@WeakestHunter43 Can you explain how does that work and how to adapt it for more than one language? And does this work on Windows 11 (PowerShell)?

Is this issue not fixed yet?

We can remove the duplication using awk tool (ChatGPT helped me to know how), and clean unneeded empty lines using SubtitlesCleanerCommand. I have included awk, SubtitlesCleanerCommand and yt-dpl into environment variable path. And using the following batch script I can get clean youtube subtitles:
@echo off

set /p "url=Enter the URL: "
set /p "lang=Enter the subtitle language (Ex: en, fr): "
yt-dlp --write-auto-subs --sub-langs %lang% --convert-subs srt -o "%%(title)s.%%(ext)s" --skip-download %url%

:: Remove duplicates and clean
for %%i in (*.srt) do (
	awk "NF==0 || !seen[$0]++" "%%~ni.srt" > temp.srt && move temp.srt "%%~ni.srt"
)
SubtitlesCleanerCommand clean --path "./" --save --suppressBackupFile --suppressWarningsFile --quiet

Note: awk may fail to read the srt file if its name contain symbol.

in mpv I use youtube-download.lua
table.insert(command, "--write-auto-subs")
change to
table.insert(command, "--write-auto-subs")
table.insert(command, "--convert-subs=srt")
this way it converts vtt to srt as it downloads it..otherwise you need to use ffmpeg to convert to srt.
d-f run /bin/bash -c "cd ~/data/mpv/download; [ ! -d fixedsubs ] && mkdir fixedsubs; for f in *.srt ; do cat $f |  awk '!/./ || !seen[$0]++' | awk 'NF{c=1} (c++)<3' > fixedsubs/$f ; done" ;show-text "fixed yt auto-generated subs"           #! [Subtitles] > [fixsrt] > fix yt auto-generatesubs
the first awk removes duplicate lines but it does leave duplicates with different timestamps which is ok since they don't overlap.  Second awk removes any blank lines that are more than one.
d-c run /bin/bash -c "cd ~/data/mpv/download/fixedsubs; rm .srt ../.srt" ;show-text "Deleted srt in Desktop download and fixedsubs directories"   #! [Subtitles] > [fixsrt] > Delete srt subs download & fixedsubs dirs
Screenshot remember the repeating lines are fine as the times don't overlap so no repeating subs along with other subs
Left side shows yt-auto-generated converted to srt with youtube-download and right side shows fixed subs

this one just once it gets like 5 or 6 subs just delete the srt subs to clean it up.
I was using python srtfix and it's awesome..but don't want users having to install python plus python path (beyond most average users ability) just for fixing yt auto-generated subs.

@mrfragger Good thought.
I tired awk "!/./ || !seen[$0]++" in my batch script but the result is different from what do you say, anyway the code I use works good.

This is what I use now
subdemofixyt.sh save to ~/.config/mpv/extrastuff
#!/bin/bash
cd ~/datampv/download
[ ! -d fixedsubs ] && mkdir fixedsubs
for f in *.srt
do cat $f |  awk '!/./ || !seen[$0]++' | awk '!NF {if (++n <= 2) print; next}; {n=0;print}' > fixedsubs/$f
done

cd fixedsubs
for i in *.srt
do sed -i .fixed -E -e 's/\.$|\. /\_ /g' -e 's/'\''/´/g' -e 's/\?/\¥/g' -e 's/([A-Za-z])-/\1~/g' -e 's/([0-9])(,)([0-9])/\1COMMACOMMA\3/g' -e 's/,/¸/g' -e 's/\(|\[/{/g' -e 's/\)|]//g' -e 's/#|: |;|<=/~/g' -e 's/!/||/g' -e 's/\(|\[/{/g' -e 's/COMMACOMMA/,/g' $i
done

[ ! -d demofontupper ] && mkdir demofontupper
for f in *.srt
do cat $f | awk '{print toupper($0)}' > demofontupper/${f%.*}.demofontupper.srt
done

[ ! -d demofont ] && mkdir demofont
for f in *.srt
do mv $f demofont/${f%.*}.demofont.srt
done

[ ! -d lowercase ] && mkdir lowercase
for f in *.fixed
do cat $f | awk '{print tolower($0)}' > lowercase/${f%.fixed.*}.lowercase.srt
done

[ ! -d uppercase ] && mkdir uppercase
for f in *.fixed
do cat $f | awk '{print toupper($0)}' > uppercase/${f%.fixed.*}.uppercase.srt
done

[ ! -d fixed ] && mkdir fixed
for f in *.fixed
do mv $f fixed/${f%.*}
done

[ ! -d srtdownloaded ] && mkdir srtdownloaded
cd  ~/datampv/download
for f in *.srt
do mv $f fixedsubs/srtdownloaded/$f
done

d-f run /bin/bash -c "cd ~/.config/mpv/extrastuff; ./subdemoytfix.sh";show-text "Fixed youtube auto-generated subs"           #! [Subtitle Fonts] > [fix online video subs] > fix yt auto-generatesubs
d-c run /bin/bash -c "cd ~/datampv/download/fixedsubs; rm demofont/.srt ; rm demofontupper/.srt ; rm fixed/.srt; rm lowercase/.srt; rm uppercase/*.srt" ;show-text "Deleted srt in Desktop download and fixedsubs directories"   #! [Subtitle Fonts] > [fixsrt] > Delete srt subs download & fixedsubs dirs

I find that Aegisub which is available from most linux distro, I believe, can solve this problem with simple click. You can embed the subtitle with ffmpeg or  yt-dlp later, or leave it as a single file.
Follow this step:

Open Aegisub
Navigate the subtitle
Select All line with shortcut Ctrl - A or Menu: Subtitle > Select All
Timing > Make times continuous > Change End
Voila, You're done!

Before:

After:


Pre-release v0.0.1 updates

Looks good code-wise! I didn't try to pull it down to run it or anything though, I assume it all still runs 👍
:dancer:


Fix documentation for $select

The documentation was incorrect (probably because it aws copied from the original NeDB and MongoDB behaviour).
Closes https://github.com/feathersjs/feathers-mongoose/issues/71

:shipit:


fix: Some privileges can't be grant to user

issue: milvus-io/milvus#38003
cause the privilege use wrong object_name_index, which makes the privilege will never be permitted

/lgtm
/approve


Remove t.Skip("TEMP_SKIP_FOR_NULL_POINTER_FAST_FAIL_TEST")

If anyone knows how to fast fail the tests, or knows why this throws null pointer errors in CircleCI only, please pick this up. When done, search and remove this line from all tests.
t.Skip("TEMP_SKIP_FOR_NULL_POINTER_FAST_FAIL_TEST")

⬆ @Bubblyworld @binocarlos @lukemarsden

REMINDER, this works fine locally, but breaks in CircleCI. I don't know why.

I think this is okay - I've run the tests a bunch locally after removing the Skip(...)s, and it runs on CI fine too.


feat(company certificate): list company certificates

Description
create a link in user menu to navigate to company certificate page
add filter and sort component
display certificates in grid fashion
Why
new feature
Issue
NA
Checklist
Please delete options that are not relevant.

[x] I have followed the contributing guidelines
[ ] I have performed IP checks for added or updated 3rd party libraries
[ ] I have created and linked IP issues or requested their creation by a committer
[x] I have performed a self-review of my own code
[x] I have successfully tested my changes locally
[ ] I have added tests that prove my changes work
[ ] I have checked that new and existing tests pass locally with my changes
[ ] I have commented my code, particularly in hard-to-understand areas

API integration is pending

Zero linter issues

@oyo conflicts resolved


?queue isn't working

Describe the bug
?queue trows a console error
To Reproduce
Steps to reproduce the behavior:

Add music to the queue
Enter ?queue
No there is no answer sometimes

Expected behavior
Normally there should be the whole music queue
Screenshots
https://i.imgur.com/0T6oh2r.png

#93


[P0] Some cards are missing their cardbacks

Summary
Pretty sure this is related to pixelFormat errors yet again, but there are no errors or warnings in the browser console to suggest it this time.
Note: This looks like a regression between 1.97.5 and 1.97.6 (seems to have worked on web 1.97.5, but we should confirm by trying old versions locally). Affects Firefox, Chrome, and Mac/Electron.
Impacted services

[x] The frontend app
[ ] The game servers
[ ] Game builds or automation scripts
[ ] Tests or Github Actions
[ ] Infrastructure (Terraform)

Steps to reproduce

Play a practice game
Hover over some specific cards in hand, like Thorn Needler or Widowmaker at a minimum.
See the cardback missing; it's just transparent.

Environment information

Operating system: Mac
Browser: Electron desktop client
Browser version: 1.97.6

Possibly related warning in browser console:
WebGL warning: bindTexture: Object `tex` is already deleted.

After testing builds of 1.97.5 and 1.97.6 locally, I think this is somehow related to the changes in these two files:
app/ui/views2/shop/confirm_purchase_dialog.js
app/ui/views2/shop/premium_purchase_dialog.js

Not yet sure how this could impact rendering cardbacks in game code, unless it's causing some problem in the WebGL context which is persisting? TBD.

I believe something went wrong during the yarn build process when packaging the app for staging yesterday. I rebuilt the app today with no changes, yet the filesize of duelyst.js is 1,404 bytes larger, and the issue is no longer present.
Will reupload the build + desktop clients and verify the issue is resolved.

Looking at the diff (after running it through js-beautify) shows some changes to code generated by glsify, which could definitely cause the behavior observed. The rebuilt app seems to be working normally. I've uploaded it to staging S3 and published new clients on the existing release.
We may want to update GLSL / glsify in the future.


Web Hook Not Firing

SETHOOK NB_6500 redis://localhost:6379/local INTERSECTS FD WHERE Bearing 340 +inf WHERE Bearing -inf 20 FENCE DETECT enter OBJECT { "type": "Feature", "properties": { "IntID": 6500, "Longitude": -111.840065942233, "Latitude": 33.400579713564497 }, "geometry": { "type": "Polygon", "coordinates": [ [ [ -111.840065942232997, 33.400579713564497 ], [ -111.83949460992865, 33.393726108276795 ], [ -111.840065942232997, 33.393709373780418 ], [ -111.840637274537343, 33.393726108276795 ], [ -111.840065942232997, 33.400579713564497 ] ] ] } }
Trying to create FENCE that will trigger webhook based on direction of travel using the bearing provided by GPS. All others work except Northbound.   ex.  Eastbound  70 +inf  -inf 110   Westbound  250 +inf   -inf 290    Southbound 160 +inf   -inf 190    However it will work if i exclude WHERE Bearing -inf 20  but this will only cover someone traveling  NNW to N    where Im trying to cover NNW to NNE

After looking closer at the docs I see why its not working.  I was assuming mutiple WHERE statements provided an  OR like function but  i was wrong.  Is it possible to add a statement that provides this or do I need to use the WHEREIN and list  340,341,342,343,344,345...0,1,2,3,4,5,6,7...20   and does providing a large list like  this take away from performance.

There WHEREIN with lots of options is one way. But I would be concerned with performance.
Also try out the WHEREEVAL option. It utilizes the builtin Lua interpreter.
INTERSECTS FD WHEREEVAL "return FIELDS.Bearing >= 340 or FIELDS.Bearing <= 20" 0 ...

I recommend experimenting with both

I have a hook set like this
{"ok":true,"hooks":[{"name":"geoId_0","key":"fleet","endpoints":["kafka://<IP_ADDRESS>:9092/geofishing","kafka://<IP_ADDRESS>:9093/geofishing","kafka://<IP_ADDRESS>:9094/geofishing","kafka://<IP_ADDRESS>:9095/geofishing","kafka://<IP_ADDRESS>:9096/geofishing","kafka://<IP_ADDRESS>:9097/geofishing","kafka://<IP_ADDRESS>:9098/geofishing"],"command":["INTERSECTS","fleet","FENCE","DETECT","enter","OBJECT","{\"type\":\"Feature\",\"properties\": {\"site\":\"Airport\",\"siteid\":\"10\", \"cid\": \"153\"},\"geometry\":{\"type\":\"Polygon\",\"coordinates\":[[[80.877656,26.761911],[80.877656,26.763941],[80.881754,26.763941],[80.881754,26.761911],[80.877656,26.761911]]]}}"]}],"elapsed":"0s"}
and when i do set fleet 123123 POINT 77.10205078124999 28.554369806808154

am i doing something wrong ?

Oh my bad... I will correct it...
if i have to detect  "enter,exit" on object  e.g below ``

r1.execute_command('SETHOOK', savetile,
'kafka://<IP_ADDRESS>:9092/geofishing,kafka://<IP_ADDRESS>:9093/geofishing,kafka://<IP_ADDRESS>:9094/geofishing,kafka://<IP_ADDRESS>:9095/geofishing,kafka://<IP_ADDRESS>:9096/geofishing,kafka://<IP_ADDRESS>:9097/geofishing,kafka://<IP_ADDRESS>:9098/geofishing',
'INTERSECTS','fleet','FENCE','DETECT','enter','exit','OBJECT',hookline)

its giving me error..

''enter,exit',' this sovled it...


Add rake console command with handy things

This adds a console for debugging ActiveShipping code and also more importantly the APIs of the carriers.
In the past when I'm trying to figure out something about a carrier's API I have used an IRB session with a bunch of stuff loaded and had to copy-paste a bunch of things from tests and load a bajillion files.
This PR adds a rake console command based on pry that has a bunch of handy things pre-defined and included:

ActiveShipping and testing fixtures and credentials modules included
An equivalent to the reload! method from 'rails console` that reloads all of the active_shipping code
A bunch of predefined carrier objects using the local test credentials from the remote tests
A method for pretty-printing XML strings (unfortunately too commonly needed)

@kmcphillips @wvanbergen cc @RichardBlair because he could have used this in his weekend debugging session

Is this necessary? Is pry --gem not sufficient?

The test helpers and credentials are really necessary to get anything done. This has already helped me significantly in looking into UPS surepost stuff. The pre-created carrier objects and reloading also save a lot of time.
The reason Active Shipping needs this but other gems don't as much is all the API integrations require lots of valid frippery just to test them, and they need a lot of manual testing because the docs are often insufficient, this saves the effort of finding and including all that frippery.
https://www.youtube.com/watch?v=8To-6VIJZRE

You make a strong argument.

:+1:

:+1: - a good development and test environment would really make working on this gem a lot easier. Maybe document it in the wiki/README/CONTRIBUTING?


fix 342 - error when specifying sequential struct layout of a type

Fix #342 - Emits the attributes correctly in F# Interactive, adds some testing (I chose an existing F# Interactive test to add it to somewhat at random)

LGTM


data folders for params model checkpoint for pycharm don't work

model.load_state_dict(torch.load(args.params))
gets
IsADirectoryError: [Errno 21] Is a directory: 'data/emissary'
Do these model checkpoints need to be updated for the pycharm version? Do you happen to have the samples?

Hi, Jim.
Can i get more info on where exactly this is happening? Nothing in here should depend on what IDE you like to use.

I'm trying to run reamp.py with the data that's in the data folder:
python reamp.py architectures/fc_32_32_32.json data/emissary /Users/jamsden/temp/Guitar1_1.wav --output /Users/jamsden/temp/Guitar1_1-processed.wav
torch.load(args.params) fails on data/emissary. That data looks old, i.e., the model hasn't been updated using the new PyTorch framework.

I thought I could get reamp.py working on your sample model before creating my own amp model. I have an early '60's Tremolux amp I want to capture.

Closing this as I ended up ripping up the old way of handling data in v0.2.


Fix a broken screenshot and remove another

Fix screenshots has been broken

I signed it!

This sample has been deprecated/archived (check README for more information on newer samples related to this technology).
As recommended by GitHub, we are closing all issues and pull requests.


Export .dds in the correct pixel format

Description

I would like the possibility to export Cubemaps to ".dds" by preserving the DDS pixel format.
For example, in the current version if I save a cubemap that has an R16G16B16A16_FLOAT format, the saved .dds file will have a fourcc of 808540228, corresponding to BC6H, instead of 113, corresponding to R16G16B16A16_FLOAT.

Environment

RenderDoc version: 1.8
Operating System: Windows 10
Graphics API: DX12/DX11

Your description is incorrect. Listing the fourcc as a decimal number is a confusing way to present it, but 808540228 corresponds to the code DX10 (0x30315844) indicating that the dx10 dds header is present to specify the format. RenderDoc certainly doesn't encode anything to a block compressed format on export as that would be complete nonsense.
DDS files are always exported as the exact format with no encoding, if the pixel format isn't representable the export fails.


Add support for .set().

https://github.com/jonschlinkert/remarkable#set

#15 solves this. Pass a function(md) to the 2nd parameter, and push the plugin to the md object.


Can pathogen load plugins in the bundle directive before loading the scripts in vim root directive?

It seems the pathogen always try to load the scripts in the vim/ftplugin first, before loading the scripts in vim/bundle/myplugin/ftplugin.
I am wondering if there is a way that I can load the latter first? Because I always want to put the customized configurations for the third-party plugin in the vim/ftplugin  directive. But if the corresponding scripts of plugin hasn't been loaded yet, my configuration won't succeed, due to problems like "undefined variable", etc.

Vim provides a directory your customized configurations: vim/after/ftplugin.

Oh thank you so much! Can't believe I didn't know this before!


Create Nikhil202003.json

Fixes Issue

Changes proposed

Check List (Check all the applicable boxes)

[X ] My code follows the code style of this project.
[ ] My change requires changes to the documentation.
[ ] I have updated the documentation accordingly.
[ ] All new and existing tests passed.
[ X] This PR does not contain plagiarized content.
[X ] The title of my pull request is a short description of the requested changes.

 I would like my LinkFree profile be updated

You have conflicts in your file, please resolve. This is usually because your fork is not up to date.
You may wish to have a look at the following Docs and video: https://linkfree.eddiehub.io/docs/how-to-guides/editing

@harshsingh32 and @iamthecloverly thank you for reviewing the Pull Request.  However as there are conflicts these are not the final changes and your approvals will be dismissed by GitHub after these are resolved.  Plus these temporary approvals create more notifications and potentially confusion for the author.

We look forward to seeing your Profile on LinkFree, but as there is no recent activity from you to make the changes suggested I am going to close this pull request.
We hope you will raise a new pull request to create your Profile and when you do, you might want to take a look at the following Docs which will help with the suggested changes the maintainers have proposed:
Quickstart: https://linkfree.io/docs/quickstart
Editing your Profile: https://linkfree.io/docs/how-to-guides/editing


Fixed Register page input component issue

:white_check_mark: Build TheIdServer 8.0.2-20.build.3846 completed (commit https://github.com/Aguafrommars/TheIdServer/commit/8fe75234f1 by @)

Can you open an issue and explain what was the bug please

#1272


create an onboarding document and list of tasks.

Pitch the new people
list background articles to read.


game-rs: 0.1.3 -> 0.2.0

Description of changes

https://github.com/Amanse/game-rs/releases/tag/v0.2.0
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[ ] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

I have a unfree dependency as a propagated build input, but the license of the program itself is MIT, what should i keep my license so as to not run unfree code for people who have it disabled? It was allowed previously in the init commit but I later found out about the mistake and apologise for this.


Dependabot config

Checklist

[ ] Update changelog in CHANGELOG.md.

Going with a ping. Was in here already creating a helm3 branch.


Crossref API integration

Build integration with the Crossref's API to scan for related identifiers

on hold since Crossref is part of the COKI Observatory Network's available tables see issue #77

closing this one as it will be part of the work that comes out of #77


The component doesn't re-render to show the updated value of a counter after the first increment

The component doesn't re-render to show the updated value of a counter after the first increment when using the global store but works fine on subsequent increments and this only happens when the initial value is set to 0, otherwise works just fine. link to sandbox https://codesandbox.io/s/legend-state-9b6pln?file=/src/App.js.
Am I missing something here?

import { observable } from "@legendapp/state";
import { observer } from "@legendapp/state/react";

const store = observable({
  legend: {
    count: 0
  }
});

const App = observer(function App() {
  const count = store.legend.count.get();

  return (

{count}

      <button onClick={() => store.legend.count.set((c) => c + 1)}>add</button>

  );
});

That is a bug! I've just written a test for it and fixed it, and will release an update tomorrow (it's 2am here in New Zealand). It's caused by poor ordering of operations in the set function where it's checking whether it's a primitive or not before it computes the function (so it thinks it's not a primitive because it's a function).

Okay got it. thanks

Ok, fixed in versions 0.23.2 and 1.0-rc.1. Thanks for the report!


Remove unnecesary error log msg when opening a SHM port  [8030]

When trying to open a SHM port (read exclusive), the open_port function logs an error if the port is already opened. This should be avoided because FastRTPS automatically tries another port number until a free port is found. Therefore, errors must not be logged to the user, errors in open_port are reported through exceptions that, in this case, are catched and filtered by the upper layer.

Build status:

Linux
Mac
Windows


Use status cache for resource readiness probes

When deployment engine waits for resource to became ready,
it should use ScheduledResource.Status() rather than Status method of
the underlying resource. The difference is that for ScheduledResource,
the status may be cached or set externally. For example, this
happens when Wait() times out. Without this change, the goroutine
created by Wait() will not recognize it and will continue running and
querying resource status in endless loop until deployment ends

This change is

Still described situation desires for a test (but this sounds that it would be quite tricky).

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


raftstore: support batch ingest

cherry-pick #9686 to release-5.1

What problem does this PR solve?
Problem Summary:
Ingest costs more time than other operations. It will cause high latency of apply-wait.
What is changed and how it works?
What's Changed:
To reduce the impact on apply-wait latency, this PR collect multiple ingest requests and ingest them at a time.
Related changes
Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)

Side effects

Performance regression

Consumes more CPU
Consumes more MEM

Release note

support batch ingest in raftstore

/cc @Little-Wallace @gengliqi

/run-all-tests

/run-all-tests

/run-all-tests

/test

/merge

/merge


Adding SQL Assessment extension as a recommended extension

Let's update the SQL Assessment extension as a Microsoft recommended extension. Thank you!

@alanrenmsft let me know if I need to provide you with something to get this done.


intro to additional material

https://formal-methods-mpi.github.io/Workshop.jl/stable/5_additional/intro/
could there be an intro to foundation revisited if this section is kept there just do it appears in the bullet list?

just FYI I ahve no idea what Automatic Differentiation means even when looking at that page - so I'm skipping it  - just noted they were the 'Main' again is the ouputs - and it's likely I wouldn't get them either (see #92)

Thanks for pointing this out, I will add "Foundations revisited" to the intro.


[Powertoys Run Feature Request] Reorder Plugin Results

Description of the new feature / enhancement
Allow for changing the order in which types of results are presented in powertoys run. Manually change the priority of the plugins so that certain ones appear at the top.
Scenario when this would be used?
This would optimise workflow and allow users to prioritise the plugins they use more frequently by having them at the top of the results list.
Supporting information
No response

Chances are this would be the solution here.  https://github.com/microsoft/PowerToys/issues/14495 where most is off by default.
Custom weighting our thoughts could cause a few more quirks than we want.  https://github.com/microsoft/PowerToys/issues/3169 was i believe the tracking item but looks like the original author closed it out.
would #14495 work?

/needinfo

I'm not sure #14495 would be the best solution. The use case I'm considering is being able to move the single result plugins like shell and websearch to the top of the list so that multiple results for apps and files can be accessed later if needed. This would also prevent the 'collisions' that occur between the URI handler and websearch where URI often takes priority.

I'd also appreciate this very much to have more control over how my search results look like without having to remove stuff from the global search results.

Flow Launcher solves it by having Priority number next to each plugin. The greater number, the higher the related search results will appear.
By default the value is 0, and it's greyed out. Increase it to increase priority. Set it to a negative number to get it below. It's not that complicated

It does work, but you need really weird settings - to consistently get open window results above application results, you need a score of ~200. Just setting it to 1 (and the application launcher to 0) doesn't work, even though you'd expect it to. I suspect there is some weighting based on relevance, but without diving into the guts it is hard to say.

It does work, but you need really weird settings - to consistently get open window results above application results, you need a score of ~200. Just setting it to 1 (and the application launcher to 0) doesn't work, even though you'd expect it to. I suspect there is some weighting based on relevance, but without diving into the guts it is hard to say.

This has been my experience as well. Just got it to work by setting program weighting to -50 and Window Walker to +45


Persist the 1st report of worker

I would like to see that the 1st report is persisted in the mongodb just like the last report in work_stop()

Hi @ecebuzz !
Interesting idea, can you explain why you'd want to see it?

I found that during the runtime of short jobs whose execution time is less than the report interval. Mongodb queries for those jobs will not show any results until the very end when those short jobs finish and stop. Perhaps it is not the recommended way to query runtime information of the jobs. Perhaps I should change to query redis instead of mongodb.

It seems the change does not quite help. Does mrq dashboard query redis for the job and worker status?


Add layout selection

Add layout algorithm selection
Add all layouts from Network Layouts
Add a fixed position SFDP layout

Codecov Report

Merging #81 (3e17396) into master (c1a3e76) will increase coverage by 0.13%.
The diff coverage is 95.32%.

@@            Coverage Diff             @@
##           master      #81      +/-   ##
==========================================
+ Coverage   86.80%   86.93%   +0.13%
==========================================
  Files          11       11
  Lines         432      467      +35
==========================================
+ Hits          375      406      +31
- Misses         57       61       +4

Impacted Files
Coverage Δ

src/PowerPlots.jl
100.00% <ø> (ø)

src/plots/plot.jl
56.00% <50.00%> (-1.70%)
:arrow_down:

src/layouts/common.jl
93.61% <89.28%> (-6.39%)
:arrow_down:

src/layouts/SFDP_fixed_layout.jl
98.43% <98.43%> (ø)

src/experimental/experimental.jl
51.51% <100.00%> (ø)

src/layouts/kamada_kawaii_layout.jl
96.10% <100.00%> (ø)

src/core/configuration.jl
77.77% <0.00%> (-11.12%)
:arrow_down:

src/core/utils.jl
88.46% <0.00%> (-6.14%)
:arrow_down:

... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c1a3e76...3e17396. Read the comment docs.


fix(#3640) : errorFallbackSrc and loadingFallbackSrc props  to image component

Closes #3640
📝 Description
⛳ Current behavior (updates)
Fallback displays same fallback image for error and loading state
🚀 New behavior
errorFallbackSrc and loadingFallbackSrc props are added to display different fallbacks for both states.
💣 Is this a breaking change (Yes/No):
I don't know

Summary by CodeRabbit

New Features

Introduced errorFallbackSrc and loadingFallbackSrc properties for improved image handling during loading and error states.
Enhanced user experience by displaying relevant placeholder images instead of broken icons during loading errors.

Documentation

Updated documentation to reflect the new properties and provide clearer usage instructions for the Image component.

will be handled in #3664


load_rsa_private_numbers() missing 1 required positional argument: 'unsafe_skip_rsa_key_validation'

Hi,
When trying to decrypt Master Key with Domain PVK I have the following error
Traceback (most recent call last):
  File "/path/bin/pypykatz", line 8, in <module>
    sys.exit(main())
  File "/path/lib/python3.10/site-packages/pypykatz/__main__.py", line 89, in main
    helper.execute(args)
  File "/path/lib/python3.10/site-packages/pypykatz/dpapi/cmdhelper.py", line 151, in execute
    self.run(args)
  File "/path/lib/python3.10/site-packages/pypykatz/dpapi/cmdhelper.py", line 217, in run
    dpapi.decrypt_masterkey_file_with_pvk(args.masterkeyfile, args.pvkfile)
  File "/path/lib/python3.10/site-packages/pypykatz/dpapi/dpapi.py", line 354, in decrypt_masterkey_file_with_pvk
    privkey = PVKFile.from_file(pvkfile).get_key()
  File "/path/lib/python3.10/site-packages/winacl/dtyp/wcee/pvkfile.py", line 72, in get_key
    return RSAPrivateKeyBlob.from_bytes(self.keyblob).get_key()
  File "/path/lib/python3.10/site-packages/winacl/dtyp/wcee/rsaprivkey.py", line 52, in get_key
    return default_backend().load_rsa_private_numbers(numbers)
TypeError: Backend.load_rsa_private_numbers() missing 1 required positional argument: 'unsafe_skip_rsa_key_validation'

My command is
pypykatz dpapi masterkeypvk AppData/Roaming/Microsoft/Protect/S-1-5-21-xxx-xxx-xxx-500/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx ntds_capi_xxxxx-xxxx-xxxx-xxxx-xxxxxxxxx.pvk

Can't identify if problem comes from pypykatz or winacl
Do you have a solution ?

I don't have a solution as I

Can't reproduce the issue
Checking the documentation on load_rsa_private_numbers here doesn't mention any required arguments besides that supplied.


Issues with SPM packages and Container static shared instance

We've been using Factory for a while in our project trouble free. However, we've just begun to run into issues whereby all of our mocked dependencies setup from unit tests are being ignored.
This issue started when we made the final migration to remove CocoaPods in favour of SPM. Previously we were utilising development CocoaPods for all our modules, whereas these are now local SPM packages. A trimmed down example of one of our SPM module packages looks like this:
let package = Package(
    name: "ModuleA",
    defaultLocalization: "en",
    platforms: [
        .iOS(.v15),
    ],
    products: [
        .library(
            name: "ModuleA",
            targets: ["ModuleA"]
        ),
        .library(
            name: "ModuleAMocks",
            targets: ["ModuleAMocks"]
        ),
    ],
    dependencies: [
        .package(
            url: "https://github.com/hmlongco/Factory.git",
            exact: "2.2.0"
        )
    ],
    targets: [
        .target(
            name: "ModuleA",
            dependencies: [
                .product(name: "Factory", package: "Factory")
            ],
            path: "Framework/Sources",
            plugins: [
            ]
        ),
        .target(
            name: "ModuleAMocks",
            dependencies: [
                .byName(name: "ModuleA"),
            ],
            path: "Tests/Sources/Mocks",
            plugins: [
            ]
        ),
        .testTarget(
            name: "ModuleATests",
            dependencies: [
                .byName(name: "ModuleA"),
                .byName(name: "ModuleAMocks")
            ],
            path: "Tests/Sources",
            exclude: [
                "Mocks",
            ],
            plugins: [
            ]
        ),
    ]
)

And the DI container defined in the ModuleA target:
public final class ModuleAContainer: SharedContainer {
    public static var shared = ModuleAContainer()
    public var manager = ContainerManager()
}

And then a mock extension on the container defined in the ModuleAMocks target
public extension ModuleA {
    static func setUpMocks() {
      // shared.depdency.register { mockedDependency }
      // ...
    }

Each unit test class then calls the required ModuleA.setUpMocks() etc in the setup.
It's worth also mentioning that each modules suite of unit tests run fine when run in isolation, i.e. directly from the SPM package. However, the issues are back in the main project, both on the test plan comprising the suite of units tests which are part of the main projects test target and the test plan comprising of all of the SPM modules unit tests.
Despite the tests setting all the mocked dependencies, when the projects main code resolves those dependencies during a test execution, a new instance of the container is created, not the statically allocated one created by the test fixture calling setUpMocks().
Given that static variables are scoped to a module in Swift (by design) and SPM packages are compiled separately to one another, I'm struggling to see how it's possible to use Factory in the context of a modularised project which utilises SPM for those modules because having a statically allocated shared container is a fundamental requirement of a container.
Has anyone else encountered similar issues and if so how did you approach this problem? Thanks!

I believe we're experience a similar outcome while using Factory as an SPM dependency across multiple modules.
While running unit tests in main project the container is being initialized more than once and instead of using mock it uses real implementation from the ModuleA.
I've created a demo project including two SPM modules(which runs fine in isolation), but then linked together into main target produce multiple container recreation.
If ModuleBMocks is removed from ProjectTests's linked frameworks then unit tests working properly in the main target(testBuilderExample).
Here are two projects, one containing ModuleBMocks dependency and another doesn't.
Please suggest any workaround here, thanks!
ProjectSuccess.zip
ProjectFail.zip

I am also having a similar situation, could you please help with this?


ur: use index based slicing

This removes the clippy::indexing-slicing from the CI as we use the .get method on Vec and slices as we are using unwrap on it anyway.
Using index-based slicing lets the user of the crate remove the bounds check (once the code is proven to work correctly) and is a bit more readable, for code that we should handle the error where the index is off we should use get instead.

I agree with the reasoning, thanks for the cleanup! Would you mind squashing the commits? I prefer if every commit has a passing CI.

@dspicher done


Issue 14/product detail page

#14 Add simple product detail page

Awesome @Oppedijk , LGTM! Ship it 🚀


Implement frontend for setting a GitHub PAT

Blocked on implementing backend for this

See here for what we'll want to use in the backend: https://github.com/mergestat/fuse/blob/main/migrations/20220710232941_service_authentication_table.up.sql#L18-L20

@german-mergestat FYI the Validate button maybe can make a client-side only API request? Something simple just to confirm the token works with the GitHub API?
Also - while you are here, can you swap the position of GitHub Authentication and General? Also, can you make it GitHub Authentication (not Github Authentication)


Coral Dev mini board forcing fastboot mode and reflashing failed

Description
I have a dev mini board that doesn't work. So I'm trying forcing fastboot and reflashing procedure as instructed. However, I tried using Ubuntu 20.04 installed on Win10, flashing failed. I'm wondering how exactly it should be done and does it only work on true Linux machine? as in the docs, it says the flashing procedure works for Mac and Linux only. Thanks for the help!
Click to expand!
Issue Type
Build/Install
Operating System
Ubuntu
Coral Device
Dev Board Mini
Other Devices
No response
Programming Language
Python 3.8
Relevant Log Output
No response

Hi, Here are some log details with Linux Machine.
bash enable_lk_fastboot.sh
**********************************************************************
Unable to connect to debug board, check connection or manually toggle GPIO.
**********************************************************************
/tmp/tmp.A2frymb1B7 ~/Downloads/excelsior-eagle-20201210233645
INFO: pySerial version: (3.5)
INFO: Use config file: /tmp/tmp.A2frymb1B7/dl_addr.ini
INFO: Waiting to connect platform...

At this point, please use a wire to connect the KPCOL0 test point to ground, as shown in figure 2. Ideally, use a female-to-male jumper wire, with the female end attached to pin 39 on the GPIO header (ground), and then hold the male end firmly on the test point.

While maintaining a firm connection with the jumper wire, carefully connect the USB data cable to the board's USB OTG port.

INFO: Got /dev/ttyACM0
INFO: Connect brom
INFO: Loading file: lk.bin
INFO: Send lk.bin
INFO: Jump da
~/Downloads/excelsior-eagle-20201210233645

If everything goes well, you should be able to see above logs and please follow the rest of the steps as described in the docs.

Thanks for the prompt reply. I was actually asking if the reflashing process MUST be done on true Linux machine. Anyway, I tried it on a Linux machine, and it worked. So I'm good for now. Please close this thread. Thanks!


Question: Is this library thread-safe?

Hello, I basically want to know if the library is thread-safe or not (I mean within the same application; inter-process will obviously be safe)

Hi there @TheDonsky, this library is not thread safe in the way that using it at the same time from multi-threads will bring problems.
Anyway I'm not 100% if all 3 implementations would fully work if you use both APIs (low level and high level) from the non-main (non-UI) thread (i.e. the thread running with the main message loop of events).

Hi again, huge thanks for the information and the fast response.
I'll add a 'global clipboard lock' to my project.
As a side note, so far I've tested only on windows without any UI and will need to make it work on Linux too. Mac OS is not my priority for now. And in both those cases, I have a manually created thread managing the Window with no problems so far (If I'm not mistaken, you can even have multiple event threads for both). Will probably run into a few issues when I finally get to build anything MacOS-related, though.


RAFT_vaiq_int8 model support

Failed op:

https://github.com/nod-ai/SHARK-Turbine/issues/636

Before fixing expand op, we cannot get any other fail op signature information. The following cmd is blocked by expand.
torch-mlir-opt -convert-torch-onnx-to-torch ./RAFT_vaiq_int8.default.torch-onnx.mlir
./RAFT_vaiq_int8.default.torch-onnx.mlir:2045:13: error: failed to legalize operation 'torch.operator' that was explicitly marked illegal
    %2041 = torch.operator "onnx.Expand"(%2002, %2040) : (!torch.vtensor<[1,2,?,?],f32>, !torch.vtensor<[?],si64>) -> !torch.vtensor<[],f32>
            ^
./RAFT_vaiq_int8.default.torch-onnx.mlir:2045:13: note: see current operation: %7506 = "torch.operator"(%7142, %7505) <{name = "onnx.Expand"}> : (!torch.vtensor<[1,2,?,?],f32>, !torch.vtensor<[?],si64>) -> !torch.vtensor<[],f32>

Next failures are

onnx.Resize (which should be fixed by torch-mlir PR# 3013)
A matrix multiplication that is getting half-quantized. Looking into this now, but it's going to be difficult to fix, since the second operand is getting dequantized before several shape manipulations occur.

failed to translate executables
failed to translate executables
failed to translate executables
RAFT_vaiq_int8.default.onnx.torch.mlir:1644:13: error: 'func.func' op exceeded stack allocation limit of 32768 bytes for function. Got 401408 bytes
    %1601 = torch.aten.quantize_per_tensor %1596, %float5.000000e-01, %int0, %int12 : !torch.vtensor<[1024,7,7,1],f32>, !torch.float, !torch.int, !torch.int -> !torch.vtensor<[1024,7,7,1],!torch.qint8>
            ^
RAFT_vaiq_int8.default.onnx.torch.mlir:1644:13: note: called from
    %1601 = torch.aten.quantize_per_tensor %1596, %float5.000000e-01, %int0, %int12 : !torch.vtensor<[1024,7,7,1],f32>, !torch.float, !torch.int, !torch.int -> !torch.vtensor<[1024,7,7,1],!torch.qint8>
            ^

Maybe we can use --iree-llvmcpu-fail-on-out-of-bounds-stack-allocation=false. but I'm not sure if that is the best option. I've been sitting on iree-compile for a while now.

Related issue in iree: https://github.com/iree-org/iree/issues/17455.

@IanWood1 PR to fix: https://github.com/iree-org/iree/pull/17574


Capturing video

Hi, could you maybe add an example for capturing video like here https://github.com/MicrosoftDocs/SimpleRecorder?

This may be what you're looking for:
https://github.com/robmikh/displayrecorder
Although it uses Media Foundation directly instead of Windows.Media.Transcoding.

thanks :)

No problem! Let me know if you have any questions. Although I should warn you that I'm by no means a Media Foundation expert, my background is in the capture side of things.
I'm also on holiday until the new year, so responses may be delayed.


Stuck fiber on 5th mag bright star in tile 3827

One of the (stuck?) fibers in petal 1 landed close to a 5th mag star and likely ruined the spectra of nearby fibers, and it is marked as unsure in QA.
https://www.legacysurvey.org/viewer/?ra=343.8032&dec=8.8160&layer=ls-dr9&zoom=15&catalog=tmp5i8cmxjy

it s FIBER=862, LOCATION=1338.
same thing as for https://github.com/desihub/desisurveyops/issues/155 which happened on 20231021, this fiber "became stuck" on Oct. 17:
           TIME           LOCATION STATE     POS_T       POS_P     MIN_P    MAX_P    MIN_T      MAX_T      EXCLUSION
------------------------- -------- ------ ------------ --------- --------- ------- ---------- --------- ----------------
2022-09-21T17:00:39+00:00     1338      0          0.0       0.0 1.8821365   182.0 -193.84338 190.84338 81e666b0d9f8a699
2023-10-17T17:00:48+00:00     1338      2 -0.084694706 143.42325 1.8821365   182.0 -193.84338 190.84338 81e666b0d9f8a699

@schlafly : would it make sense to ask Dustin to run is "nudging-tiles" code on the remaining, unobserved main tiles, and then decide on the results of those -- ie how many fibers are expected to fall on bright stars -- to take action or not?
btw, that makes me think that for the mainbright 5th pass, this nudging would not have been applied to tiles "twinning" the 5th maindark tiles observed before Dustin last ran is code, no? (sorry I don t have clear recollection here...)

@schlafly @rongpu @araichoor please confirm if you'd like me to flag petal 1 as bad and reprocess this tile.  It is currently on-hold in an "unsure" QA state.

Thanks @sbailey , yes, please mark P1 as bad and reprocess.  Sorry!

Here is what the diagnostic plot looks like for this tile.  This plot is showing "nudge space" in arcseconds, and the shaded circles are stars around stuck positioners.  You can see that there is a gray background shading, corresponding the 5th-mag star, and the nudge we arrive at is one at the largest radius.  The code reports that the nudge we chose has "badness" of 10, where a badness of 1 is our target flux.

For reference, a more typical diagnostic plot would look like this -- there is some white background = "okay" within the nudge radius.

I have flagged petal 1 as bad for tile 3827 night 20231104 expid 203453 and reprocessed it.  Back to @schlafly for re-QA.

Marked as good, closing!


PYCS2 agent on linux server - /pysc2/run_configs/platforms.py ValueError: max() arg is an empty sequence

Hello, i am trying to run simple PYCS2 agent on linux server, and it throws me this error.
could someone tell me what it means and what can i do to fix it?
Traceback (most recent call last):
File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
"main", mod_spec)
File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/bin/agent.py", line 112, in
app.run(main)
File "/home/xkraji11/.local/lib/python3.5/site-packages/absl/app.py", line 274, in run
_run_main(main, argv)
File "/home/xkraji11/.local/lib/python3.5/site-packages/absl/app.py", line 238, in _run_main
sys.exit(main(argv))
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/bin/agent.py", line 98, in main
run_thread(agent_cls, FLAGS.map, FLAGS.render)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/bin/agent.py", line 74, in run_thread
visualize=visualize) as env:
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/env/sc2_env.py", line 132, in init
self._setup((agent_race, bot_race, difficulty), **kwargs)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/env/sc2_env.py", line 185, in _setup
self._launch(interface, player_setup)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/env/sc2_env.py", line 205, in _launch
self._sc2_procs = [self._run_config.start()]
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/run_configs/platforms.py", line 87, in start
data_version=data_version, **kwargs)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/run_configs/lib.py", line 104, in start
return sc_process.StarcraftProcess(self, **kwargs)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/lib/sc_process.py", line 67, in init
exec_path = run_config.exec_path(game_version)
File "/home/xkraji11/.local/lib/python3.5/site-packages/pysc2/run_configs/platforms.py", line 73, in exec_path
build_version = max(int(v[4:]) for v in os.listdir(versions_dir)
ValueError: max() arg is an empty sequence

It looks like it isn't finding any versions of SC2 where you're pointing it at. Did you install the game correctly? Admittedly it could use a better error message.

Better error message in 5339c11 .


Kotlinx serializer annotation are ignored

The OpenAPI generator doesn't seem to respect kotlinx serializer annotations, such as @Transient, and fields marked with such are included in the spec and swagger example.

Tegral OpenAPI uses Swagger Core under the hood, which in turn uses Jackson, which uses a different set of annotations. There are two ways to go around this:

Either also add Jackson annotations to your models in addition to kotlinx.serialization annotations. Not great, but it should work.
Find some way to tell Swagger Core how kotlinx.serialization annotations should be interpreted. No idea how to do this but I'll figure something out.


fix merge pipeline

Please make sure the code is following contribution guidelines in CONTRIBUTING.md

[ ] : This PR has a corresponding issue open in the Repository.
[ ] : Approach is signed off on the issue.

Why merging with single approval?
merging this so that in case some other change breaks something then we can catch it.
without this change out merge pipeline is currently broken


System.InvalidOperationException: Attempt to write to a read-only page

This is a big issue I encounter once in a while using BrightstarDB.
When doing some
context.DeleteObject(myobject);
context.SaveChanges();

I will be able to call it dozens and dozens of times, but at some point (often when I delete 5 to 10 objects in a row) this error will came up :

System.InvalidOperationException: Attempt to write to a read-only page
à BrightstarDB.Rdf.NTriplesParser.Parse(TextReader reader, ITripleSink sink, String defaultGraphUri)
à BrightstarDB.Server.GuardedUpdateTransaction.Run()
BrightstarDB Error: 400 : Error Processing Transaction System.InvalidOperationException: Attempt to write to a read-only page
à BrightstarDB.Rdf.NTriplesParser.Parse(TextReader reader, ITripleSink sink, String defaultGraphUri)
à BrightstarDB.Server.GuardedUpdateTransaction.Run()
à BrightstarDB.Server.StoreWorker.ProcessJobs(Object state)
Exception thrown: 'BrightstarDB.Client.BrightstarClientException' in BrightstarDB.dll

Passed this point, any entity from the type of the entity i want to delete will turn undeletable, every attempt ending with the same error message. Yet i'm still able to create, modifiy, delete other types of entities, this one turn into a dead end.
The only two solutions then are :

To wipe out the entire database and restart from scratch.
(After doing step 1 ) To edit the model of the object and add a Property DeletionRecquired, and hide from user every entity with this property set to true (and obviously edit whole program to handle this case)

When you delete some objects in a row do you call each time context.SaveChanges()?
Did you try to delete these objects and after call context.SaveChanges only once?
It sounds like a cross-access. But still, it shouldn't break definitely the database.

From the exception you get I'm guessing that the store type you are using is the Rewriteable store, but can you just confirm that.
If the error isn't transient (i.e. it persists across restarts of the application/service) then it sounds like a corruption in one of the indexes, especially if it is related to one specific type of entity. That's definitely a big problem that I need to try to fix. I'll take a look, but this could be really hard to track down unless we can work together to find some sort of reproducible steps (which I'm guessing might be hard to do!).

Yep i'm using a Rewritable store

I'm working on a bug reproduction project, I'll keep you informed of my progress (and of any other bug I would find)

So far I haven't succeded in recreating this specific bug in my bug reproduction project.
But i found another one :
When i create multiple entities in a row, if i try to delete one i get the good old exception  :

Added item does not appear at given index '0'

https://drive.google.com/open?id=0B98ZSzSdzS7URjc0cnRleFdxWTQ

Okay I finally reproduced the bug !
https://drive.google.com/open?id=0B98ZSzSdzS7UM3ZmUml6OW93aXM
You just have to run this for a few minutes, then stop it, comment out the method DataCreationRoutine() and you should have the error about Readonly Page

Any updates about this bug ?

Sorry, I've been snowed under with work recently and haven't had spare time to take a proper look at this.

Is this still an issue?  I just encountered the problem that persisted across application restarts on Azure.  I published new code with latest B* develop branch and it seems to have gone away.

... but I must also mention that I have added consolidation to my rewrite store on application startup.  So that definitely did happen.


process HCP fmri on hippunfold results

Hi hippunfold expert:
I'm trying to extract fmri signal from HCP dataset. But HCP only include MNI space fmri data. I tried to register the fmri data back to native space and then project it onto the hippocampus subfields. i'm not sure if this introduces some bias of bold signal. Is there a best way to process HCP fmri data ? Can you kindly gave some advice about this problem ?
Thank you for your help!
best regards
carlos

Hi Carlos,
The easiest way to make sure your fMRI data is aligned to HippUnfold is simply to run HippUnfold on data that is also in space-MNI152. I believe that in HCP these data are rigidly aligned to MNI, and so there will still be a lot of inter-individual variability in hippocampal shape, which HippUnfold should be able to detect.
I hope that helps,

Closing, but let me know if you have further issues and i can reopen!


Dataset search API update:

New parameters needed to support https://github.com/Sp2000/colplus/blob/master/docs/EDITORIAL-TOOLS.md#main-editorial-dashboard

sortBy=KEY,CREATED,MODIFIED,SIZE
format=ACEF,DWCA
catalogue=PROV,SCRUT
code=BACTERIAL,BOTANICAL,CULTIVARS,VIRUS,ZOOLOGICAL
type=NOMENCLATOR, TAXONOMIC, NATIONAL, PERSONAL, OTHER

Update Dataset class properties:

releaseDate -> released
trusted -> catalogue=PROVISIONAL, SCRUTINIZED
size (taken from last import, not stored in dataset table)

Source for provisional and scrutinized should also be included

and https://github.com/Sp2000/colplus-backend/commit/8f6fac52e36c9106ec81befc03dd35dec1d16ce0


Make the target easier to override in plugins

Right now snakemake reads it's target directly out of the targets_by_analysis_level dict in the snakemake call. It would be better to do this before plugins run, placing the chosen target in a target entry that can be easily overriden in plugins.

Resolved by #362, where the snakemake plugin reads the "snakemake_target" entry in the config.


Wrote stopwatch.js

Closes #2
Guys, I finished the stopwatch.js. I did not come up with the functions myself (so it was a bit quicker of course),
but I do understand how it works (I guess that is great instead of just copy and paste without knowing what it means).
And I made also lots of comments within the JS file, it is just how I study on new things, dont mind them.
So please, check it out! @abelRoland @mametur

Oh guys, I forgot to write debugger!

For me is not mandatory the debugger is more for when we are testing the code, after the code is done and works is not so important (my opinion) . The console.log is nice to have, but it’s possible to add on a finishing process, then we add the same style of message in all the files 😉. Thanks to remember that.


Remove ol.raster namespace

The next and penultimate part of typedef move.
The namespace is only used currently for 2 typedefs and 1 enum, so it serves little purpose.
So the 2 typedefs are renamed in the ol namespace, and the enum moved to rastersource.js
@ahocevar if this looks familiar, it's much as you proposed in https://github.com/openlayers/ol3/commit/a096c3dae6b187e2226e1a8fffbd794540ff0b20. The main difference is that you renamed the enum to ol.source.RasterOperationType. I prefer ol.RasterOperationType so it's consistent with ol.RasterOperation. And in any case, the sub-namespaces will be disappearing as part of conversion to modules.
I generated apidocs, and the refs all look ok.

now rebased to account for #5427. As ol.raster.Pixel has now been removed, the ol.raster namespace now makes even less sense.

Thanks @probins.


Update circle.yml

Changes Unknown when pulling 26f30a2d71244b44b224e5a51ebc824c9b6adb48 on levi-rs-patch-1 into ** on master**.

Changes Unknown when pulling 26f30a2d71244b44b224e5a51ebc824c9b6adb48 on levi-rs-patch-1 into ** on master**.


Infinity height increase

if iframe body>div has margin-top:1vh it runs Infinity height increase
ex:
iframe:
body>div {   width: 98vw;   margin-left: 1vw;   margin-top: 0;   margin-bottom: 1vh; }

Have you tried playing with the tolerence option?
I expect you have somehow created a page that is sized via CSS to always be
101% the size of the page and hence the loop.
If you can create a simple test case I'll have a bit more of a look at this.
On Fri, Jun 5, 2015 at 5:14 PM, Alex Kanunnikov<EMAIL_ADDRESS>wrote:

if iframe body>div has margin-top:1vh it runs Infinity height increase
—
Reply to this email directly or view it on GitHub
https://github.com/davidjbradshaw/iframe-resizer/issues/218.

--
David J. Bradshaw )'<EMAIL_ADDRESS>'The people we think are a little weird are the ones who change the world'

Simon Sinek

Closed due to no response.


why isn't examples/tx/balances_transfer.eg.ts failing?

Should be failing given that the free balance runs reference the same rune instance (hence, the > assertion between the two should fail).

Because the two are run sequentially (not in parallel), the Runs are cleaned up in the interim, clearing the memoization and causing it to run fresh the second time around. This behavior should be valid for anything that is idempotent (and not time-dependent).
Anything that is time-dependent should be represented as such within Rune, which will mean that no matter if it is cleaned up or not, the behavior will remain correct. In other words, rather than leaving the blockHash parameter as defaulting to the latest block (at the time of execution), it should default to a subscription to the latest block.
The way we handle certain things are are not time-dependent but are also not idempotent (for example, createDevUsers) is still to be determined. Likely, such Runs will not be cleaned up until the corresponding Rune is GC'd.


steps to diagnose lack of content in logs

I've just deployed an ekanite instance, and am not seeing any log data in the system. I'm not sure whether the issue is the configuration of my client, its connectivity to ekanite, or ekanite's ability to parse the log entries being sent to it.
I've verified that the host/port for ekanite is reachable from the client system, and enabled the -diag endpoint, but without documentation showing what the diagnostic JSON should look like it's hard to know what to look for
Here's what the initial chunk of the diagnostic output looks like:
{
  "bleve": {
    "bootDuration": 12030,
    "indexes": {}
  },
  "cmdline": [
    "/usr/local/bin/ekanited",
    "-datadir",
    "/data/index",
    "-tcp",
    ":5514",
    "-queryhttp",
    ":8080",
    "-retention",
    "168h",
    "-diag",
    ":9951"
  ],
  "ekanite": {},
  "engine": {
    "queriesRx": 5
  },
  "input": {},
  "memstats": {
    "Alloc": 1269032,
    "TotalAlloc": 1269032,
    "Sys": 4165632,
    "Lookups": 38,
    "Mallocs": 8128,
    "Frees": 417,
    "HeapAlloc": 1269032,
    "HeapSys": 2686976,
    "HeapIdle": 737280,
    "HeapInuse": 1949696,
    "HeapReleased": 0,
    "HeapObjects": 7711,
    "StackInuse": 458752,
    "StackSys": 458752,
    "MSpanInuse": 25080,
    "MSpanSys": 32768,
    "MCacheInuse": 3472,
    "MCacheSys": 16384,
    "BuckHashSys": 3055,
    "GCSys": 169984,
    "OtherSys": 797713,
    "NextGC": 4473924,
    "LastGC": 0,
    "PauseTotalNs": 0,
    "PauseNs": [
      0,
      0,
      0,

After than its a ton of 0 lines for PauseNs, and then PauseEnd with a ton of 0s, and then a bunch of memory information objects
Is "input": {}, significant? The key thing I'm trying to determine at this point is whether ekanite is receiving any messages, and then how many of those it's successfully parsing. Is the empty input section telling me it hasn't received anything? Would it look any different if ekanite was receiving messages but failing to parse them? If ekanite was receiving messages but failing to parse them, would I be able to get a look at what the messages it's receiving look like anywhere and/or what the parse error is?

Hi -- as mentioned in the README (though it's at the bottom, I admit) this project is not actively maintained.
https://github.com/ekanite/ekanite/issues/98
I'm going to archive it to prevent any future confusion.


⚠ Bofrost.de has degraded performance

In f7984c0, Bofrost.de (https://www.bofrost.de) experienced degraded performance:

HTTP code: 200
Response time: 1569 ms

Resolved: Bofrost.de performance has improved in d5fc18d.


placing a mechanical turret on top of a really fast fan in a merger gearbox

https://paste.ee/p/a2myV
it flies a bit then crash

wrong mod


i18n(fr): Translation of cli-reference.mdx

What kind of changes does this PR include?

Translated content

Description
Translated cli-reference.mdx content to french

Thanks for this @Elib27, this page had a few changes recently, could you please update the translation? https://github.com/withastro/docs/commits/main/src/content/docs/en/reference/cli-reference.mdx?since=2023-01-24T11:32:29.000Z


支持1.19.x

如题

First, this is an issue and is wrongly submitted as a feature request.
Second, this issue tracker is English only, not everyone in the world speaks another language.


bug(Mat-Tab): Clicking a half-visible tab doesn't work with non-default animationDuration

Reproduction
See the video in attachment
Steps to reproduce:

setup a mat-tab with:
{ provide: MAT_TABS_CONFIG, useValue: { animationDuration: 1 } }
click a tab that is only half-visible.

Expected Behavior
Tab should jump into view AND become selected.
Actual Behavior
The tab jumps into view, BUT it doesn't become selected.
Environment

Angular: 11.2
CDK/Material: 11.2
Browser(s): chrome
Operating System (e.g. Windows, macOS, Ubuntu):  windows & Ubuntu
bug-2021-02-12_16.14.43.mkv.zip

I can sort of see it happening here: https://stackblitz.com/edit/angular-9sk3pv?file=src/app/tab-group-dynamic-example.ts. What I think is happening is that the we start scrolling the header on focus which is around the same time as mousedown. Since the element is being moved out from under the pointer before the click is finished, it sometimes gets interruped.

weird, if I try the stackblitz, the animation has the default duration

animationDuration only affects the animation of the content, not the one in the header.

ah, well, we use it in combination with NoopAnimationsModule instead of BrowserAnimationsModule.
If I add that to the stackblitz I reproduce the issue we have:
https://stackblitz.com/edit/angular-9sk3pv-px8i8g?file=src%2Fmain.ts


Rename for targetAmountX in CalcGreatestMatchingRatio, should just be reserveX

targetAmountX makes no sense since it has nothing to do with a target amount.
The ratio of targetAmount0 to targetAmount1 can be seen as a targetRatio but this doesn't make sense for amounts.

👍


Use D8 for DS18B20 prevents boot

I just discovered that using D8 (GPIO 15) toconenct a DS18B20 with the recommended pull-up prevents boot. I haven't investigated much further, but it seems that GPIO 15 is involved in bootmode selection at start-up. Maybe the wiki could be updated with this information: https://github.com/rvdbreemen/OTGW-firmware/wiki/Using-DS18B20-sensor-with-ESP-firmware

@fhennies thanks for reporting the issue. I removed D8 from the documentation.


Explicitly state what security: [] means

It's currently not clear from the spec what security: [] means. It's stated that it's allowed, but it doesn't say what it does. Obvious guesses are that it either authorizes everything, or that it authorizes nothing; but it's not stated which.
As per https://github.com/OAI/OpenAPI-Specification/issues/3938, I'm told that the answer is "authorize everything"; so this PR seeks to make that explicit.
On the other hand, if that answer is incorrect, please let me know what [] in fact means, and I'll see if I can update this PR accordingly.
See also: #3994

As @rvedotrc correctly points out

If one is going to define AND/OR for zero operands, then conventionally one defines AND-of-nothing to be true, and OR-of-nothing to be false.

This would indicate that an empty array is not a synonym for an array containing an empty object, but rather "no way to authenticate here".
On the other hand 3.1.0 states

This definition overrides any declared top-level security. To remove a top-level security declaration, an empty array can be used.

I'd rather remove/undefine/nullify a top-level security declaration with security: null.

Additional observations:
https://spec.openapis.org/oas/latest.html#schema "In the following description, if a field is not explicitly REQUIRED or described with a MUST or SHALL, it can be considered OPTIONAL."
In https://spec.openapis.org/oas/latest.html#fixed-fields , security is not marked as required / must / shall. Therefore, it's optional. If security not provided (absent? null?), then what is the behaviour? We are not told. Again, obvious guesses are "deny all" and "allow all", but it would be nice not to have to guess.

If security is not provided (because it's absent? because it's present, but null?), then what is the behaviour?

My guess: if security is not provided in the OpenAPI Object, this means "we don't tell you", and the client can assume nothing. If it is present with value null, this means "we tell you that we don't tell you", and the client again can assume nothing.

If it is present with value null

AFAICT, the word null appears precisely once in the 3.1.0 spec, and it's not in this context. Therefore, either this field cannot (currently) be null, or it can be null (presumably on account of being OPTIONAL), but in which case it would be helpful to explain that OPTIONAL means "can be present, but null".
and/or perhaps OPTIONAL means "can be absent".
Also, if present-but-null is allowed, then is that different from being absent? How about in the operation.security case?
I detect a can of worms. :-/

To summarise what we've got so far, from my point of view:
Optionality:

The meaning of OPTIONAL is not well-defined. Obvious guesses are that it means "may be absent" and/or "may be present, but null"; it would be helpful to explain which of these three is in fact the case.
If an OPTIONAL field may be present-but-null (see point 1), we should define whether or not that null overrides higher values (e.g. in the case of Operation.security). This could be defined at the schema level (i.e. add it just after the explanation of OPTIONAL here), or (more messily) for each applicable field.
Perhaps it would be helpful clearly to state, for each optional field, what the meaning is if the field is absent / present-but-null (see point 1). But here the can of worms is getting even bigger.

The security field:

It would be helpful to define what absent / present-but-null means (at the OAS level)
It would be helpful to define what absent / present-but-null means (at the Operation level)
It would be helpful to define what [] means (at either level)
AFAICT, the meaning of [] is currently not defined; I can see why people would assume it to me "authorize all", mostly because it "looks like" [{}] (they're both empty, right?); but using the normally accepted rules of boolean set logic, it would mean "deny all".

So ... should I update the PR contents so as to explain that security: [] means "there is no way to authorize"?

So ... should I update the PR contents so as to explain that security: [] means "there is no way to authorize"?

In my humble opinion: yes.
Let's see wait for others to state their opinion, here or in one of the next TDC calls.
@rvedotrc do you want to dial in? The discussion is open for everyone.

1600 UTC, right? Sure :-)

Correct, 18:00-19:00 CEST 😁

For what its worth:
An empty array in the security field indicates that no security schemes are applied to the specific operation. This means the endpoint can be accessed without any authentication or authorization. Inline with (3.0.3 and 3.1.0):

To make security optional, an empty security requirement ({}) can be included in the array.

and its override use cases at both the root or operation level.
My humble interpretation of security not being present at all is simple:  You did not define any form of security scheme, so therefore the entire API is not secured in any way (perhaps this is your requirement).
I rather not go down the nullish rabbit hole unless we feel IT IS ABSOLUTELY necessary for both humans and code.

We have three cases to consider:

no security
security: []
security: [{}]

Only the third case is explicitly mentioned in the specs:

To make security optional, an empty security requirement ({}) can be included in the array.

I would not assume that 1 and 2 are synonyms of 3.

Things we do agree on (from TDC):

if security is not defined, then we assume "allow all" for all operations
if the security array has entries, one of those entries must be satisfied for a request to be authorised

The correct behaviour for an empty array is unclear. We think it is widely used to mean that there's no security applied ("allow all") - but logically it means that there are no security rules that can be fulfilled to permit access.
We do need to update wording to explain that the empty security array is ambiguous in the current spec versions.

I did some digging, and found that @webron added the "To remove a top-level security declaration, an empty array can be used." in this Swagger/OAS 2.0 commit.  I have not been able to find any further information on whether "remove" meant "default back to deny-all" or "default back to allow-all".  Since this is 2.0, which was initially developed within Swagger, I checked their site but was unable to find any further information on security: [].
Arguably, since this language was added pre-OpenAPI, it would be valid to look at how Swagger handles an empty array for insight into the original intent.  I normally dislike using Swagger as a reference point as OpenAPI is long since a distinct entity, but this has not changed since it was added for 2.0.
I do think that the fact that {} was explicitly described as a way to make security optional suggests that maybe "remove" meant deny-all, but ¯\(ツ)/¯

Changed to draft: needs to be merged after #4035 as it changes contents of table cells and will conflict with our markdown cleanup.

In today's TDC we agreed to re-try this based on @mikekistler 's "open world" view, so closing these three attempts.
@rvedotrc thank you for your contributions here- it is far from the first topic where we've had to go through multiple approaches, attempts, and PRs.  Regardless of who writes the final PR, this has helped us immensely in figuring the problem out.


LEAN-2416 Fix for "Cant read props of undefined (toggle)"

This is small fix that doesn't need extensive testing. Merging without test deployments.


Remove $ in shell source

With $ copy+paste into console doesn't work.

Can one of the admins verify this patch?

@hydroo please rebase on master

Codecov Report

Merging #387 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #387      +/-   ##
==========================================
+ Coverage   97.96%   97.96%   +<.01%
==========================================
  Files          86       86
  Lines        5957     5958       +1
==========================================
+ Hits         5836     5837       +1
  Misses        121      121

Impacted Files
Coverage Δ

...es/Search/src/details/DTK_DetailsTreeTraversal.hpp
100% <0%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 58621ef...47967a2. Read the comment docs.


Shortcuts that are a combination (f+i) are saved under the second key - if there is a collision in that second key, the callbacks can misfire

Example:
I have two shortcuts: i which will navigate my page and f+i which will add a new item to the page.
If I just press i while selected on an item on the page, it will match with the f+i shortcut because that is saved under the self._callbacks[i] even though it is a combination.

See #323


Introduce Fujifilm compressed RAW support

Regular Fujifilm camera RAWs are working fine. Compressed ones however are not working.

I don't know if this is a "apple doesn't support that, yet" issue or a "GeoTag is not doing something right" issue.  Does Finder and quick look show the compressed images?  If so it is likely a GeoTag error.  Can you send me one of the images for me to test?

Yes, you're right. I also do not have a preview when seeing those in Finder. Here I have an image in that format for you:
DSCF6542.RAF.zip

Thanks.  I also tried looking at the image using Preview.app -- it can't read the format.  I'll submit a bug/feature request to Apple requesting that it support the format.  Both Adobe Lightroom Classic and Capture One process the image correctly.

I'm going to close this as there is nothing GeoTag can do.   It is up to Apple to fix this.


Override NSOperationQueue's mainQueue() to return the main queue as an OperationQueue

Hi @gtchance - thanks for this!
Just one thought, the documentation states: "The returned queue executes one operation at a time on the app’s main thread." I'm thinking that we should also set maxConcurrentOperationCount in addition to the underlying queue.

@danthorpe, nice catch! I made the adjustment.

@gtchance great! Thanks again!


Increase minSdk to 26 (Android 8 / Oreo)

Task/Issue URL: https://app.asana.com/0/488551667048375/1206486003155266/f
Description
Steps to test this PR
Feature 1

[ ]
[ ]

UI changes

Before
After

!(Upload before screenshot)
(Upload after screenshot)

develop

#4177   👈

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @CDRussell and the rest of your teammates on  Graphite


Restic doesn't save extended attributes, in the system namespace, on Linux

Output of restic version
restic 0.17.1 compiled with go1.23.1 on linux/amd64
What backend/service did you use to store the repository?
Local file system
Problem description / Steps to reproduce

Mount an NTFS file system, containing extended attributes (not the same as NTFS streams), from OS/2, Interix, WSL, or CygWin, using NTFS-3G (I'm using ntfs-3g 2021.8.22, on Ubuntu 22.04), and run a Restic backup, on the mounted volume.
Expected behavior
I would expect that at least the system.ntfs_ea, system.ntfs_attrib_be, and system.ntfs_times , containing data such as the following, is preserved:
# file: tmp/2/
system.ntfs_times=0sfmGf1zrUygFsejQEO9TKAcbcNgQ71MoBbHo0BDvUygE=
# file: tmp/2/
system.ntfs_acl=0sAQAEgDAAAABAAAAAAAAAABQAAAACABwAAQAAAAADFAD/AR8AAQEAAAAAAAEAAAAAAQIAAAAAAAUgAAAAIAIAAAECAAAAAAAFIAAAACACAAA=
# file: tmp/2/
system.ntfs_attrib_be=0sAAAAFg==
# file: tmp/2/http!!!www.google.com
system.ntfs_ea=0sMAAAAAAFIAAuVFlQRQDf/wAAAQD9/xYAVW5pZm9ybVJlc291cmNlTG9jYXRvcgAALAAAAAAJGQAuTE9OR05BTUUA/f8VAGh0dHA6Ly93d3cuZ29vZ2xlLmNvbQAkAAAAAAgQAC5TVUJKRUNUAP3/DABWaXNpdCBHb29nbGUAAAAwAAAAAAkeAC5DT01NRU5UUwDf/wAAAQD9/xQAQSBzaG9ydGN1dCB0byBHb29nbGUkAAAAAAsQAC5LRVlQSFJBU0VTAN//AAABAP3/BgBHb29nbGU=
# file: tmp/2/CARDFILE.CRD
system.ntfs_ea=0sIAAAAAAIDwAuU1VCSkVDVAD9/wsATXkgQ2FyZEZpbGUwAAAAAAkdAC5DT01NRU5UUwDf/wAAAQD9/xMAQ29udGFpbnMgb25lIHJlY29yZAA0AAAAAAsdAC5LRVlQSFJBU0VTAN//AAACAP3/BwBXaW5kb3dz/f8IAENhcmRGaWxlAAAA

Other extended attributes, for NTFS-3G are documented, at https://github.com/tuxera/ntfs-3g/wiki/Using-Extended-Attributes

Actual behavior

These attributes don't exist, when browsing the mounted Restic repository:
getfattr: Removing leading '/' from absolute path names
# file: tmp/2/
system.ntfs_times=0sfmGf1zrUygFsejQEO9TKAcbcNgQ71MoBbHo0BDvUygE=

root@DESKTOP-6HH5CK9:~# getfattr -n system.ntfs_times /tmp/1/tags/Understand/latest/tmp/2/
/tmp/1/tags/Understand/latest/tmp/2/: system.ntfs_times: No such attribute

root@DESKTOP-6HH5CK9:~# getfattr -n system.ntfs_acl /tmp/2/
getfattr: Removing leading '/' from absolute path names
# file: tmp/2/
system.ntfs_acl=0sAQAEgDAAAABAAAAAAAAAABQAAAACABwAAQAAAAADFAD/AR8AAQEAAAAAAAEAAAAAAQIAAAAAAAUgAAAAIAIAAAECAAAAAAAFIAAAACACAAA=

root@DESKTOP-6HH5CK9:~# getfattr -n system.ntfs_acl /tmp/1/tags/Understand/latest/tmp/2/
/tmp/1/tags/Understand/latest/tmp/2/: system.ntfs_acl: No such attribute

root@DESKTOP-6HH5CK9:~# getfattr -n system.ntfs_ea /tmp/2/CARDFILE.CRD
getfattr: Removing leading '/' from absolute path names
# file: tmp/2/CARDFILE.CRD
system.ntfs_ea=0sIAAAAAAIDwAuU1VCSkVDVAD9/wsATXkgQ2FyZEZpbGUwAAAAAAkdAC5DT01NRU5UUwDf/wAAAQD9/xMAQ29udGFpbnMgb25lIHJlY29yZAA0AAAAAAsdAC5LRVlQSFJBU0VTAN//AAACAP3/BwBXaW5kb3dz/f8IAENhcmRGaWxlAAAA

root@DESKTOP-6HH5CK9:~# getfattr -n system.ntfs_ea /tmp/1/tags/Understand/latest/tmp/2/CARDFILE.CRD
/tmp/1/tags/Understand/latest/tmp/2/CARDFILE.CRD: system.ntfs_ea: No such attribute

Do you have any idea what may have caused this?

Did restic help you today? Did it make you happy in any way?

I'm kinda perplexed, by the whole thing, to be fair.
If I create a new repository, add one of my test files to it, and compare the output of getfattr -d, to ```restic cat tree``, I see this:
root@DESKTOP-6HH5CK9:/mnt/e# ~/restic_0.17.1_linux_amd64 cat tree 45be638d:/tmp/1 -r TestRepo/ --insecure-no-password
repository 911fce87 opened (version 2, compression level auto)
found 3 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
[0:00] 100.00%  3 / 3 index files loaded
{"nodes":[{"name":"OPENDOC.OD","type":"file","mode":511,"mtime":"2010-04-04T19:22:47+01:00","atime":"2010-04-04T19:22:47+01:00","ctime":"2010-04-04T22:08:47.140625+01:00","uid":0,"gid":0,"user":"root","group":"root","inode":47,"device_id":64512,"size":16104,"links":1,"content":["39177b01afa03852a7cd94c37853b5ca6dca4ae7450e3892616541257b2e3fcd"]}]}
root@DESKTOP-6HH5CK9:/mnt/e# ~/restic_0.17.1_linux_amd64 cat tree 45be638d:/tmp/1/OPENDOC.OD -r TestRepo/ --insecure-no-
password
repository 911fce87 opened (version 2, compression level auto)
found 3 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
[0:00] 100.00%  3 / 3 index files loaded
path tmp/1/OPENDOC.OD: not a directory
root@DESKTOP-6HH5CK9:/mnt/e# getfattr -d /tmp/1/OPENDOC.OD

I can definitely see the pseudo(?) EA, if I do this, though:
root@DESKTOP-6HH5CK9:/mnt/e# getfattr -n system.ntfs_ea /tmp/1/OPENDOC.OD
getfattr: Removing leading '/' from absolute path names
# file: tmp/1/OPENDOC.OD
system.ntfs_ea=0sKAAAAAAFGgAuVFlQRQDf/wAAAQD9/xAAT3BlbkRvYyBEb2N1bWVudA==

I wonder if Linux, and/or FUSE has some sort of mechanism, for dynamically generating xattrs, on-the-fly, without registering them into some sort of master data structure?
The most information, that I can find, is a post, on a Samba-related mailing list (https://lists.samba.org/archive/samba-technical/2016-March/112751.html), from 2016, suggesting trying to mount a volume with -o streams_interface=windows, or  -o user_xattr (which seems to be a synonym, for the default mode of operation (-o streams_interface=xattr)), which doesn't seem to make a difference, when trying to fish for xattrs, using the technique that Restic, and getfattr -d use...
I guess the best approach, to working around this, would be adding some sort of --always-include-xattr option, to explicitly try to include one, or more of them, rather than trying to add a platform/FS-specific hack, into Restic, just to grovel for these xattrs?

I wonder if Linux, and/or FUSE has some sort of mechanism, for dynamically generating xattrs, on-the-fly, without registering them into some sort of master data structure?

The list and get xattr operations are simply forwarded to the FUSE filesystem. So it's trivial for the fs to hide attributes or make a mess. If the list xattr operation does not return the xattrs, then restic can't see it and back it up.

I guess the best approach, to working around this, would be adding some sort of --always-include-xattr option, to explicitly try to include one, or more named xattrs (from the user's argument), rather than trying to add a platform/FS-specific hack, into Restic, just to grovel for these xattrs?

Letting restic by default blindly try to access random xattrs is out of question. And I'm always rather hesitant to add new options as those tend to pile up and interact in unforseen ways. I also don't recall having seen such an option anywhere.
Is there really no way to make ntfs-3g simply return those xattrs when restic tries to backup a file?

That's pretty fair, in terms of adding functionality, to Restic, since the responsibility for exposing the metadata is at another VFS layer level - even if it does mean a lack of data integrity, when backing-up/restoring files.
For what it's worth, the NTFS-3G wiki has the following to say:
A few extended attributes in the system name space are used to give access to NTFS internal data which cannot be accessed through Linux standard methods. They can be accessed with any setting of the option streams_interface.

The program ntfscp.c and the shell command ntfscp.sh, available in [tools.zip](https://jp-andre.pagesperso-orange.fr/tools.zip), demonstrate how files can be copied with their internal data.

Being in the system name space, they are not returned when the extended attribute list is queried, and consequently, not copied by standard file management tools.

(The extended attribute mapping described below only applies since ntfs-3g-2010.5.16AR.1) To get the extended attribute copied by standard tools, ntfs-3g has to be compiled with the ./configure option --enable-xattr-mappings and a mapping to an alternate extended attribute in the user name space has to be defined. The standard file defining the mapping is XattrMapping in the hidden directory .NTFS-3G on the root of the NTFS partition. An alternate location may be defined by the mount option xattrmapping=path where path is either a full path on a previously mounted volume or a path relative to the root of the same NTFS partition.

Doing so, the mapped extended attributes will be copied, for instance by tar with option --xattrs, by cp with option --preserve=xattr, by rsync with option -X, etc.

Please note : having standard tools blindly copy attributes may lead to conflicts (ACLs, symbolic links, OIDs, etc.) or undesired behavior (data compression).

The extended attribute mapping file should have a line per mapped attribute with two fields separated by a colon. The first field is the system name, the second is the user name

Without locally/upstream patching it, for every user that wants to use Restic, together with NTFS-3G, or manually hunting down the source to the version that ships with Ubuntu, and rebuilding it, I'm unsure if there's a way to forcing it to always expose the attributes, in a reliable way, for every possible situation (including read-only volumes).
I am kinda sceptical, of the claim of not being possible to access the system namespace, given that I'm sure that GNU TAR can access SELinux attributes via it, on Linux distributions that default to enabling it, but I've filed a bug with NTFS-3G, to see what they say.
I should probably also see how Paragon's NTFS driver handles xattrs, but I wouldn't be surprised, if it doesn't lack support, if it's anything like their APFS driver, from my experience.

I haven't tested this, yet - but, it sounds like it might be possible to specify a mapping file, containing the following, at mount time (ntfs-3g -o xattrmapping=/root/ntfs-attrs.txt), with a fairly recent version of NTFS-3G:
system.ntfs_attrib:user.ntfs_attrib
system.ntfs_times:user.ntfs_times
system.ntfs_attrib_be:user.ntfs_attrib_be
system.ntfs_reparse_data:user.ntfs_reparse_data
system.ntfs_acl:user.ntfs_acl
system.ntfs_dos_name:user.ntfs_dos_name
system.ntfs_times:user.ntfs_times
system.ntfs_times_be:user.ntfs_times_be
system.ntfs_crtime:user.ntfs_crtime
system.ntfs_crtime_be:user.ntfs_crtime_be
system.ntfs_object_id:user.ntfs_object_id
system.ntfs_efsinfo:user.ntfs_efsinfo
system.ntfs_ea:user.ntfs_ea

Not ideal, but if it works, I'll report back...

Aha!
If I mount my disk image, with ntfs-3g -o ro -o xattrmapping=/home/tyson/ntfs-attrs.txt /dev/mapper/loop0p1 /tmp/1/ (i.e. it will not work, with a relative path, to the ntfs-attrs.txt, or "combined style" arguments, to -o (i.e.  -o ro xattrmapping,ntfs-attrs.txt), it looks like their workaround kinda works, with the caveat of data encoding issues, in the attribute values:
root@DESKTOP-6HH5CK9:/home/tyson# xattr -l /tmp/1
user.ntfs_attrib:
0000   36 00 00 00                                        6...

Traceback (most recent call last):
  File "/usr/bin/xattr", line 33, in <module>
    sys.exit(load_entry_point('xattr==0.9.7', 'console_scripts', 'xattr')())
  File "/usr/lib/python3/dist-packages/xattr/tool.py", line 196, in main
    attr_value = attr_value.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xad in position 2: invalid start byte
root@DESKTOP-6HH5CK9:/home/tyson# getfattr -d /tmp/1/
getfattr: Removing leading '/' from absolute path names
# file: tmp/1/
user.ntfs_acl=0sAQAEkIwAAACcAAAAAAAAABQAAAACAHgABQAAAAAEGACYAR8AAQIAAAAAAAUgAAAAIAIAAAAEGACIABIAAQIAAAAAAAUgAAAAIAIAAAAEFACIABIAAQEAAAAAAAEAAAAAAAQYAL8BHwABAgAAAAAABSAAAAAgAgAAAAQUAL8BHwABAQAAAAAABRIAAAABAgAAAAAABSAAAAAgAgAAAQIAAAAAAAUgAAAAIAIAAA==
user.ntfs_attrib=0sNgAAAA==
user.ntfs_attrib_be=0sAAAANg==
user.ntfs_crtime=0sFlitXjbTygE=
user.ntfs_crtime_be=0sAcrTNl6tWBY=
/tmp/1/: user.ntfs_dos_name: Operation not permitted
/tmp/1/: user.ntfs_ea: No such attribute
/tmp/1/: user.ntfs_object_id: No such attribute
user.ntfs_times=0sFlitXjbTygGKi2QmQtPKAZiyayZC08oBiotkJkLTygE=
user.ntfs_times_be=0sAcrTNl6tWBYBytNCJmSLigHK00Ima7KYAcrTQiZki4o=

Obviously, it doesn't expose them, via the system.* attributes, in a nice way, but I guess it's better than nothing, for trying to back up their contents?

Looks like Restic will at least try to access the mapped attributes, and will complain about ones that fail through:
root@DESKTOP-6HH5CK9:/home/tyson# ~/restic_0.17.1_linux_amd64 -v --insecure-no-password -r /mnt/e/Archive3 backup /tmp/1/
open repository
repository f8b018ac opened (version 2, compression level auto)
found 3 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
using parent snapshot 1f82f9fa
load index files
[0:09] 100.00%  126 / 126 index files loaded
start scan on [/tmp/1/]
start backup on [/tmp/1/]
scan finished in 12.112s: 16 files, 7.249 KiB
can not obtain extended attribute user.ntfs_dos_name for /tmp/1:

Files:          15 new,     1 changed,     0 unmodified
Dirs:            1 new,     2 changed,     0 unmodified
Data Blobs:     14 new
Tree Blobs:      4 new
Added to the repository: 27.771 KiB (7.577 KiB stored)

processed 16 files, 7.249 KiB in 0:13
snapshot 5cced57c saved

And, if I mount my repository, it looks like it's preserved the mapped attributes, and their contents:
root@DESKTOP-6HH5CK9:/mnt/e# getfattr -d /tmp/1/snapshots/latest/tmp/1/
getfattr: Removing leading '/' from absolute path names
# file: tmp/1/snapshots/latest/tmp/1/
user.ntfs_acl=0sAQAEkIwAAACcAAAAAAAAABQAAAACAHgABQAAAAAEGACYAR8AAQIAAAAAAAUgAAAAIAIAAAAEGACIABIAAQIAAAAAAAUgAAAAIAIAAAAEFACIABIAAQEAAAAAAAEAAAAAAAQYAL8BHwABAgAAAAAABSAAAAAgAgAAAAQUAL8BHwABAQAAAAAABRIAAAABAgAAAAAABSAAAAAgAgAAAQIAAAAAAAUgAAAAIAIAAA==
user.ntfs_attrib=0sNgAAAA==
user.ntfs_attrib_be=0sAAAANg==
user.ntfs_crtime=0sFlitXjbTygE=
user.ntfs_crtime_be=0sAcrTNl6tWBY=
/tmp/1/snapshots/latest/tmp/1/: user.ntfs_ea: No such attribute
/tmp/1/snapshots/latest/tmp/1/: user.ntfs_object_id: No such attribute
user.ntfs_times=0sFlitXjbTygGKi2QmQtPKAZiyayZC08oBiotkJkLTygE=
user.ntfs_times_be=0sAcrTNl6tWBYBytNCJmSLigHK00Ima7KYAcrTQiZki4o=

Whilst it'd be nice, if the NTFS-3G folks would fix their code, I guess this could be considered a "documentation bug", in terms of bear traps, that folks will be unaware of, especially considering that NTFS EAs are becoming more prevalent, with the rise of Windows Subsystem for Linux?

Interestingly, Restic can pull the security.selinux attributes, when I archive a file, on my Android phone, using the prebuilt ARM64 Linux binaries, under Termux Ubuntu chroot:

I assume there's a different code path, for exposing SELinux metadata, deep in the wilds of the VFS infrastructure?

Some thougts...
Restic supports part of the NTFS attributes if used with a windows operating system.
For interoperability  it would be nice to translate NTFS-3Gs user.ntfs_* attributes into the equivalent NTFS attributes supported by restic on the windows.
BTW: The same makes sense for sambas security.NTACL xattr.

For interoperability it would be nice to translate NTFS-3Gs user.ntfs_* attributes into the equivalent NTFS attributes supported by restic on the windows.

Restic is not meant as a tool to losslessly convert between OS-specific file attributes. That said, if someone cares enough to implement it, then maybe. But only if the changes don't leave a mess behind and do not significantly complicate things.

user.ntfs_crtime=0sFlitXjbTygE=
user.ntfs_crtime_be=0sAcrTNl6tWBY=
[...]
user.ntfs_times=0sFlitXjbTygGKi2QmQtPKAZiyayZC08oBiotkJkLTygE=
user.ntfs_times_be=0sAcrTNl6tWBYBytNCJmSLigHK00Ima7KYAcrTQiZki4o=

The *_be lines are probably the big endian encoding of the attribute without the suffix. crtime and times also appear to be potentially redundant.

For interoperability it would be nice to translate NTFS-3Gs user.ntfs_* attributes into the equivalent NTFS attributes supported by restic on the windows.

Restic is not meant as a tool to losslessly convert between OS-specific file attributes. That said, if someone cares enough to implement it, then maybe. But only if the changes don't leave a mess behind and do not significantly complicate things.

Sure, lossless conversion will not really be possible. But converting the most important things may be helpful. Therefore this suggestion. I will observe and possibly make a pull request.

Sure, lossless conversion will not really be possible. But converting the most important things may be helpful. Therefore this suggestion. I will observe and possibly make a pull request.

I'm currently a bit cautious about adding new windows-specific features. restic 0.17.0 added xattr and acl support on Windows and required a ton of follow-up bugfixes.

Interestingly, it looks like reading from the security namespace works, after running some tests:
root@DESKTOP-6HH5CK9:~# touch "Test"
root@DESKTOP-6HH5CK9:~# setcap cap_net_bind_service+ep Test
root@DESKTOP-6HH5CK9:~# xattr -l Test
security.capability:
0000   01 00 00 02 00 04 00 00 00 00 00 00 00 00 00 00    ................
0010   00 00 00 00                                        ....

root@DESKTOP-6HH5CK9:~# ~/restic_0.17.1_linux_amd64 --insecure-no-password -v backup Test -r /mnt/e/Archive3/
open repository
repository f8b018ac opened (version 2, compression level auto)
found 5 old cache directories in /root/.cache/restic, run `restic cache --cleanup` to remove them
no parent snapshot found, will read all files
load index files
[0:14] 100.00%  154 / 154 index files loaded
start scan on [Test]
start backup on [Test]
scan finished in 16.992s: 1 files, 0 B

Files:           1 new,     0 changed,     0 unmodified
Dirs:            0 new,     0 changed,     0 unmodified
Data Blobs:      0 new
Tree Blobs:      1 new
Added to the repository: 370 B (302 B stored)

processed 1 files, 0 B in 0:18
snapshot 505ab81d saved
root@DESKTOP-6HH5CK9:~# ls /tmp/1/snapshots/latest/Test
/tmp/1/snapshots/latest/Test
root@DESKTOP-6HH5CK9:~# xattr -l /tmp/1/snapshots/latest/Test
security.capability:
0000   01 00 00 02 00 04 00 00 00 00 00 00 00 00 00 00    ................
0010   00 00 00 00                                        ....

I kinda forgot about the whole Capabilities subsystem, until I stumbled across an article about Tivoli Storage Manager, and remembered that BedRock Linux uses them, as an internal implementation detail...


Implementation of JupyterBasics class that creates a part of a report

Class is to implement in en.py module that is a reporting component guideing through basic usage of Jupyter Notebook.

Done


OLS UI website fix

I noticed the from the picture that the design is on a mobile screen and so have followed suite. I made two variations with difference in the background for the announcement. Feedbacks are highly welcome!

Same as #460, I will close this PR. Thanks a lot


<EMAIL_ADDRESS>hash check failed

Scoop Configuration

{
  "last_update": "2022-10-16T15:26:25.4651259+09:00",
  "scoop_repo": "https://github.com/ScoopInstaller/Scoop",
  "scoop_branch": "master"
}

Fixed in https://github.com/ScoopInstaller/Versions/commit/af6abb258f44061e93334ce9af2018cc1757daef


Generate button does not save the map

Describe the Bug with repro steps
Currently "Generate" does not save the map as well. Keep "Save" as a separate gesture and update "Generate" to automatically save the map when generating the XSLT.
Screenshots or Videos
No response
Version
v2.63
Additional context
No response
AB#25148609

Tested and "generate" button works without "saving" (with .draft file generated). This was probably fixed.