CSV validation: is it RFC 4180 compliant?

Hi,
if I run
frictionless validate "https://gist.githubusercontent.com/aborruso/3b675b529da264be83c7e75e26d2ee26/raw/e18c3cbbb5217acc430c950c3f5ae622eb52e661/tmp.csv"

I have "valid" result.
But in RFC 4180 I have "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes."
In the input I have the cell LISTA CIVICA | "SAINT-OYEN - TRAVAILLI EUNSEMBLO" and this is not a valid RFC 4180 cell.
It should be "LISTA CIVICA | ""SAINT-OYEN - TRAVAILLI EUNSEMBLO""".
Then the question is: should frictionless validation be RFC 4180 compliant? If yes, this seems to me a bug.
Thank you

Thanks @aborruso, we will investigate

I think it's similar to the issue we discussed a few month ago, I guess in this case the software also can't figure out a data publisher intention regarding quotes and for software it's RFC4180 compliant but with a content non intended by an author. What I can't understand in this case is why validation doesn't fail with some dimension error like extra/missing-cell if the parser reads cells incorrectly. @shashigharti is analysing it atm

We use csv.reader underhood and the default value for CSV.reader dialect is:
quoting = QUOTE_MINIMAL
quote_char = "
So it parses the cell correctly even if there is "double quote" in the cell  which is not enclosed in "".
Here we use sniffer to predict the default dialect params.
https://github.com/frictionlessdata/framework/blob/main/frictionless/formats/csv/parser.py#L33
So I think it is parsing correctly even if the csv doesn't have the double quotes around the field text that has quote char in it.

@shashigharti @roll then currently frictionless can mark as valid a CSV file that is not RFC 4180 compliant. Am I right?
It would be very useful to add a validate option - --rfc4180 - to make frictionless able to run a full RFC 4180 check.

yes. And I tried using default config as well(python csv reader) but it doesn't complain even if RFC 4180 doesn't apply.
Yes it would be useful. WDYT @roll ?

I don't think it's possible ATM -- we just "don't see" these quotes (missing or provided) as we operate with a byte stream before csv.parser and a cell stream after csv.parser

BTW, RFC 4180 - https://www.ietf.org/rfc/rfc4180.txt
   5.  Each field may or may not be enclosed in double quotes (however
       some programs, such as Microsoft Excel, do not use double quotes
       at all).  If fields are not enclosed with double quotes, then
       double quotes may not appear inside the fields.  For example:

       "aaa","bbb","ccc" CRLF
       zzz,yyy,xxx

   6.  Fields containing line breaks (CRLF), double quotes, and commas
       should be enclosed in double-quotes.  For example:

       "aaa","b CRLF
       bb","ccc" CRLF
       zzz,yyy,xxx

If fields are not enclosed with double quotes, then double quotes may not appear inside the fields
Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes

TBH I don't see any MUST requirements here. Usually SHOULD (and of course MAY) are not considered as obligatory
Also, Frictionless' comment on this RFC - https://specs.frictionlessdata.io/tabular-data-resource/#csv-file-requirements

Hi @roll and as usual, thank you for your time.

TBH I don't see any MUST requirements here. Usually SHOULD (and of course MAY) are not considered as obligatory

In the CSV RFC you have a lot of "should", because is a RFC and not a final definition of the standard.
Just to give an example of another "should" for a very important element: "Each line should contain the same number of fields throughout the file."

Also, Frictionless' comment on this RFC - https://specs.frictionlessdata.io/tabular-data-resource/#csv-file-requirements (for example, the file we're discussing is not RFC 4180 valid anyway as it's not comma separated)

In this comment I read "CSV files MUST follow RFC 4180 with the following important exceptions allowed". What about to add the exception about double quotes inside cells?
Once again, thank you

Hi @aborruso, thanks again.
I agree with you that it would be nice to have a feature that could check quotes inside cell but python built in csv.parser library which we use, does not  raise any error but tries to adhere to RFC 4180 standard by adding quotes silently while parsing.
For example:
# CSV File
# Name;Address;Country
# Liam;Quebec;Canada
# Charlotte;Denver "International";USA
# Amelia;Platz des;Berlin
# Elijah;"Platz ""des""";Germany

usecase 1 - with inferred dialect
with open('test-csv-1.csv', newline='') as csvfile:
    dialect = csv.Sniffer().sniff(csvfile.read(1024))
print("Dialect", dialect.__dict__, "\n")

print("+++ Data +++")
# delimiter = ';' quotechar='"'
with open('/home/shashi/Desktop/data-encoding/test-csv-1.csv') as csvfile:
    reader = csv.reader(csvfile, dialect = dialect)
    for index,row in enumerate(reader):
        print(' -- '.join(row))

Dialect {'__module__': 'csv', '_name': 'sniffed', 'lineterminator': '\r\n', 'quoting': 0, '__doc__': None, 'doublequote': True, 'delimiter': ';', 'quotechar': '"', 'skipinitialspace': False}

+++ Data +++
Name -- Address -- Country
Liam -- Quebec -- Canada
Charlotte -- Denver "International" -- USA
Amelia -- Platz des -- Berlin
Elijah -- Platz "des" -- Germany

Row 2 is not enclosed in quotes and row 4 is, but both row 2 and row 4 are read correctly.
usecase 2 - default
print("+++ Data +++")
# delimiter = ',' quotechar='"'
with open('/home/shashi/Desktop/data-encoding/test-csv-1.csv') as csvfile:
    reader = csv.reader(csvfile)
    for index,row in enumerate(reader):
        print(' -- '.join(row))

+++ Data +++
Name;Address;Country
Liam;Quebec;Canada
Charlotte;Denver "International";USA
Amelia;Platz des;Berlin
Elijah;"Platz ""des""";Germany

In usecase 2, where we use default values with delimiter = "," it reads without any error. But it reads row 2 and row 4 differently but no error.

I agree with what @shashigharti  said above, better to wait to see if a strict mode is implemented in the standard csv library or look for another library.
There is the library https://github.com/alan-turing-institute/CleverCSV that is a drop in substitute to the standard csv library and it can convert a csv to a RFC 4180 compliant one, but it also don't raise an exception when a row is not compliant to the RFC. The library says that it can detect the dialect better with messy csv files:

With our method we achieve 97% accuracy for dialect detection, with a 21% improvement on non-standard (messy) CSV files compared to the Python standard library.

The only way that I can see to detect that the csv is no compliant to the RFC 4180 would be to change the standard csv library or the clevercsv (in that case it would need to be done in the C parser https://github.com/alan-turing-institute/CleverCSV/blob/master/src/cparser.c). Other idea would be to just convert the csv to a standard compliant one using clevercsv and comparing the different csv files, but this would not be recommended for big csv files.

Ok, I close it. Consider for yourself whether to include this validation exception in the documentation.
And thank you very much


feat: add procedure image in CoC

Types of changes
Thanks for sending a pull request! Please fill in the following content to let us know better about this change.
Please put an x in the box that applies

[ ] Bugfix
[x] New feature
[ ] Refactoring
[ ] Breaking change (any change that would cause existing functionality to not work as expected)
[ ] Documentation Update
[ ] Other (please describe)

Description
add procedure image in CoC
Steps to Test This Pull Request
Steps to reproduce the behavior:

Run Dev Server
Go to http://localhost:8000/zh-hant/accounts/agreement/

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (267c525) 70.59% compared to head (20911a0) 70.59%.
 Consider uploading reports for the commit fddc160 to get more accurate results

:exclamation: Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1165   +/-   ##
=======================================
  Coverage   70.59%   70.59%
=======================================
  Files          85       85
  Lines        3394     3394
=======================================
  Hits         2396     2396
  Misses        998      998

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


README: improve English style, add motivation

I try to explain why running your tests inside a VM may be required.

Thanks for the improvements


How to get Fair Value rating ?

Hi,
Does anyone have a suggestion on how to get the Fair Value rating shown on the yahoo finance overview page, using the yfinance package?
Thanks
Martin

To get Yahoo's own "Fair Value" rating you need a premium account, which this API doesn't support.


fix: graphql format output type

Summary
Fix a type issue that may break the playground.

I was looking at that and wondered why the type is any.
We have two options:

we make the workspace_types.rs better
we change the types returned by those functions to match what returned by the other functions


Adjust leaning behaviour of vehicles

Hello,
for a simulator setup I need to create a controllable client (like the manual_control.py) with a first person camera but this camera mustn't roll on the X axis (= the horizon shall stay at the same level at all times without any rotation). My idea was to play around in the Unreal project to permit an Actor any rotation on the X axis but until now I was not able to figure it out.
Since one can observe the leaning behaviour in the curves on all vehicles my guess is that this is some parent class issue.
Does anyone have an idea how to achieve that?

I now found a solution that works for me.
First I copied one of the existing bicycles and then in the vehicle blueprint I did two things:
I put a contraint on the rotation on the Y axis (I was expecting the X axis but the longitudinal axis of the CARLA vehicles is apparently the Y axis) of the vehicle mesh AND the skeleton mesh of the rider.
This resulted in new steering issues but so it is far from perfect but so far it was the only way I figured to do that


fix: make dark mode the default for new visitors

@hendriknielaender please have a look here, i believe our page is currently still in white mode by default for new visitors ⏰
https://giphy.com/clips/sonicmovie-sonic-paramount-movie-ixeyDqK6aao6WSdvpL


Add woff2

Based on #15, so #15 should be merged first.

Really looking forward to woff2 support, any update on this? Thx

Sorry to be an ass, but is there anything holding this up? Thanks


inkscape 0.92.3 failing to build an Darwin with "Argument list too long"

This was built using nixpgks-unstable d91caac6c3e58b8a5f4721c0a6cc8f0dc3b93fd3
$ nix build nixpkgs.inkscape
builder for '/nix/store/9hv124k15gh4wdx5z3nrkbk72ja448xh-inkscape-0.92.3.drv' failed with exit code 2; last 10 log lines:
  [ 88%] Building CXX object src/CMakeFiles/inkscape_base.dir/sp-tspan.cpp.o
  [ 88%] Building CXX object src/CMakeFiles/inkscape_base.dir/sp-use-reference.cpp.o
  [ 89%] Building CXX object src/CMakeFiles/inkscape_base.dir/sp-use.cpp.o
  [ 89%] Building CXX object src/CMakeFiles/inkscape_base.dir/splivarot.cpp.o
  [ 89%] Building CXX object src/CMakeFiles/inkscape_base.dir/viewbox.cpp.o
  [ 89%] Linking CXX shared library ../lib/libinkscape_base.dylib
  /nix/store/8b0gj3bmcy5bh1zcnhn0146199b3m5kb-clang-wrapper-5.0.1/bin/clang++: line 183: /nix/store/p85m242dg1zd85ln3cg0agx0q1n0n6h0-clang-5.0.1/bin/clang++: Argument list too long
  make[2]: *** [src/CMakeFiles/inkscape_base.dir/build.make:18100: lib/libinkscape_base.dylib] Error 126
  make[1]: *** [CMakeFiles/Makefile2:695: src/CMakeFiles/inkscape_base.dir/all] Error 2
  make: *** [Makefile:130: all] Error 2
cannot build derivation '/nix/store/1dkqbwlnx1fxc9gy0vcpc9m5c0i7vbjw-system-applications.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/xii7570acgzsy42aj844wwj5hh7y4x3w-system-path.drv': 1 dependencies couldn't be built
cannot build derivation '/nix/store/93xjrg7ky7r8bxgsl00vqy22jm9wpbmw-darwin-system-18.09.git.b0826c4.drv': 1 dependencies couldn't be built
[0 built (1 failed), 0.0 MiB DL]

This happens to quite a few CMake derivations. I think it's coming from this issue:
https://gitlab.kitware.com/cmake/cmake/issues/16206
Maybe updating to CMake 3.11 will fix it?


[ci] fix last_method rake issue.

Pins rake version to 11.3.0, older versions (not sure at which point) removed last_comment and were failing. I'm not sure which of the modules we use is calling last_comment (I believe we're not). Once we do that we can maybe put in a cleaner fix.

👍

MySQL check failing due to unrelated issue (port is in use). Merging.


Monokai and monokai dark backgrounds (+workbench colors) have changed

The background colors have changed from the standard Monokai, we should change them back to the standard background and adjust workbench colors accordingly as Monokai is a very popular theme with clearly defined colors that users expect.
https://github.com/Microsoft/vscode/pull/25599
fyi @bpasero

/cc @cleidigh

@Tyriar
Totally agree - accident and oversight
I was going to continue to add to the theme , but I'm also happy to fix myself
just let me know

@cleidigh no worries :smiley: I'll do a pass through first as we're so close to the release.

@Tyriar
Cool sorry about that
I will do some pre- consulting when adding more UI colors for May

Got regular Monokai looking pretty nice so far:

Added a bunch of stuff too:

Editor widget
Peek
Git diff
Terminal
Bunch of borders
Debug toolbar

Pushed the Monokai changes to release/1.12

oh well, I guess my workbench choices wiffed...

@Tyriar your latest changes make the theme broken in some places. Examples:
Notifications:

Section Header bleeding into background:

You should also define colors for:

titleBar.activeBackground (used on macOS for the custom title)
editorGroup.background (used when you drag an editor group left or right when the background shines through)

I must also admit that the new selection color is very strong:

PS: what about "Monokai Dimmed"?

@Tyriar here is my proposal based on the work from @cleidigh with some color tweaks:
"colors": {
	"dropdown.background": "#383852",
	"list.activeSelectionBackground": "#383852",
	"list.focusBackground": "#425370",
	"list.inactiveSelectionBackground": "#23324e",
	"list.hoverBackground": "#005070",
	"list.highlightForeground": "#e58520",
	"button.background": "#1e4456",
	"editor.background": "#202025",
	"editor.foreground": "#f8f8f2",
	"editor.selectionBackground": "#49483e",
	"editor.lineHighlightBackground": "#303030",
	"editorCursor.foreground": "#c07020",
	"editorWhitespace.foreground": "#383880",
	"editorIndentGuide.background": "#505037",
	"editorGroupHeader.tabsBackground": "#222228",
	"editorGroup.background": "#1a1a1e",
	"tab.inactiveBackground": "#333340",
	"tab.border": "#000030",
	"peekView.border": "#3655b5",
	"panelTitle.activeForeground": "#ddffff",
	"statusBar.background": "#354550",
	"statusBar.debuggingBackground": "#354550",
	"statusBar.noFolderBackground": "#354550",
	"titleBar.activeBackground": "#354550",
	"activityBar.background": "#292935",
	"activityBarBadge.foreground": "#ffffff",
	"activityBarBadge.background": "#3655b5",
	"sideBar.background": "#232327",
	"sideBarSectionHeader.background": "#424250",
	"pickerGroup.foreground": "#77a5b0",
	"terminal.ansiWhite": "#ddffff",
	"notification.background": "#292935"
}

We talked through this in the standup and feel that we should not force strong colors to the workbench for the monokai theme. We can always release a Monokai theme as extension to give this a try.

@bpasero thanks for pointing that out, I actually had a chat with Kai about the strong accent color yesterday. Will look into alternatives today.

Section Header bleeding into background

This was actually intentional as I was using a limited palette.
I was going to get to Monokai Dimmed this morning.

Changed to use the much more muted contrast border color #75715E:

Also added inputOption.activeBorder.

@cleidigh it's not that they were bad, the theme actually looked pretty good. Unfortunately the background color changed which changes the feel of the already established theme.

Also covered input validation:

For dimmed I restored the background color and removed the saturation from the blue tinted colors so @cleidigh's work is preserved for the most part. As a result it's a little lighter than before but looks fairly consistent. Also added inputOption.activeBorder and focusBorder keys to use the accent blue #3655b5

@cleidigh please feel free to iterate on this if you want :smiley:

All fixes pushed to master and release/1.12, should be good to go :+1:

@Tyriar
sounds good and looks good (I would put a thumbs up here but I have no idea how to do it ;-)
I think I needed a better understanding of the current theme goals
more than anything it's practically impossible to work with colors by voice (no mouse picking)
I think I'll focus on helping finish the feeding code ;-)
thanks for patching it up

@cleidigh basically the goal is that they should just look good and consistent, without drastically changing the "feel" of the theme (keep the background, try to reuse existing colors).
The work remaining on Monokai Dimmed is filling in a bunch of keys where it may not look right. For example many colors will fallback to the default dark theme which look alright, such as the debug toolbar, but it's the grey from the dark theme not the dimmed grey. Other things like tweaking the diff/validation/etc. colors to match the token colors would also give it a more consistent and polished look.
Thanks again for the contribution! :+1:

@Tyriar
@bpasero
I know it's late in the game, maybe has to wait till May but I was just doing some debugging
and using the latest Monokai Dimmed.
I noticed the status bar is the exact same color as the list separators , this is particularly noticeable
in my screenshot with no distinction between the breakpoint header and status bar.
personally I think the status bar should have some color, maybe steel blue or whatever.  Having
no distinction from the other elements subjugates it too much IMHO

@cleidigh what do you think about this (for status bar + title bar)?

This is #20242E which is part way between the editor background #1e1e1e and the activity bar badge blue #3655b5

We could also do a new gray in between maybe the editor and side bar?

@Tyriar
I like bringing back the blue, maybe a bit lighter but same tone?
one other thing , the list focus gray might use a little more contrast with the foreground text color
HOWEVER : I am using a regular LCD TV 4 feet from my chair so that may diminish
my perception and it's certainly not a high-quality HD computer monitor so perhaps
I can't be a really good judge of this...
I like the  line separator idea also

I just looked closer #24282F is too close to the background of the sidebar
#21242f is better
you should really decide with a good monitor and the spirit of the theme

@Tyriar
what did you go with for the status bar?

I didn't end up making a change, looking at it again I think the blue is too strong, looks strange being the only color with saturation in the workbench.

I dig all the new themes in 1.12, as it gives each a more unique feeling! Excellent work. :)
Though, one thing perplexes me: why is the activityBarBadge.background a muted gray in the regular Monokai theme?
Personally, I love and use Monokai because it is very colorful, and I have honestly grown attached to that little colorful badge. Fortunately, you can customize colors on top of a theme.
I just wanted to throw this concern out there (rather than a whole new issue). I have had a harder time reading the badge due to the decreased color contrast as well (which may be a problem, and the gray background blends in with the icon without the color). I would have expected this behavior more in the Monokai Dimmed theme just by the name, hence my confusion.
Either way, great work on the updates!

@Tyriar
understandable, as you can see from comments there are so many choices and opinions!
cool thing is it's almost all configurable now. my challenge is to get the select boxes,
they are tricky little buggers - almost have it !

@mckennapsean I made the badge and focus color was the Monokai pink originally but the team was opposed to it as it was a little too bold. It's very easy to fork the theme and publish a variant though. You can also override the colors in your settings file using workbench.colorCustomizations.


Re-design modifier interface

like not, resolves, rejects

:tada: This issue has been resolved in version 1.0.0-beta.22 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


[varLib.interpolatable] Speed up working on variable fonts

Only check each glyph at its own "master" locations instead of at all master locations across the font.
Incomplete.
Fixes https://github.com/fonttools/fontbakery/discussions/4301

This is not complete yet. Running it on RobotoFlex gives this error:
Traceback (most recent call last):
  File "/Users/behdad/fonttools/./fonttools", line 11, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/behdad/fonttools/Lib/fontTools/__main__.py", line 31, in main
    runpy.run_module(mod, run_name="__main__")
  File "<frozen runpy>", line 229, in run_module
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/behdad/fonttools/Lib/fontTools/varLib/interpolatable.py", line 586, in <module>
    problems = main()
               ^^^^^^
  File "/Users/behdad/fonttools/Lib/fontTools/varLib/interpolatable.py", line 507, in main
    problems = test(
               ^^^^^
  File "/Users/behdad/fonttools/Lib/fontTools/varLib/interpolatable.py", line 194, in test
    contour.replay(stats)
  File "/Users/behdad/fonttools/Lib/fontTools/pens/recordingPen.py", line 77, in replay
    replayRecording(self.value, pen)
  File "/Users/behdad/fonttools/Lib/fontTools/pens/recordingPen.py", line 23, in replayRecording
    getattr(pen, operator)(*operands)
  File "/Users/behdad/fonttools/Lib/fontTools/pens/basePen.py", line 224, in addComponent
    glyph.draw(tPen)
    ^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'draw'

because the reduced glyphsets in this change do not necessarily have component glyphs. Let me think how to fix it.
With this change, Handjet takes "only" 30s, instead of the previous 11hr!

Now down to 2.6s...  Working on finishing the error now.

This is ready now. Handjets takes under 3s. RobotoFlex takes 30s. I'll take a look at the latter but that one has over 80 dense masters, not sure it can be sped up further.

Fixes https://github.com/fonttools/fontbakery/discussions/4301

Well, partly. :-) Cutting instances, and even saving the TTFont is still slow.


Configuration issue: how do I set a value?

Suppose I wish to use Frame Limiter, how do I specify a value for FpsLimiter
The line is
FpsLimiter=flipend
Where and how do I put the MaxFPS value?
The game is unstable at over 60FPS. I need this to test if DDrawCompat can fix the Gamma controls for Thief 2. The gamma controls don't work, even with the AllowMaximizedWindowedGamma shim

In parentheses, like flipend(60). But the default is 60 anyway, so it's fine as you put it too. You can change it also from the config overlay (Shift+F11).

It looks like the problem was the debug executable, that freezes the game or slows it down.
I am surprised to see the in-game stars are back in 0,5.0. How did you manage it? 0.4.0 didn't work with the game, 0.3.2 had no stars.
There is a whole discussion behind the actual issue: https://www.vogons.org/viewtopic.php?p=996195#p996195

It looks like the problem was the debug dll, that freezes the game or slows it down.

The debug build has LogLevel=debug set by default, but even without it it would be slower than a release build due to additional debug checks inserted by the compiler.

I am surprised to see the in-game stars are back in 0,5.0. How did you manage it?

I don't remember doing anything for that particular issue. I remember reading theories about the game using palettized textures and that causing the missing stars, but I never found any evidence of that. There was no sign of any palettized texture usage, even when support for it was enabled in DDrawCompat.
I have the GOG version of the game, not sure what patches are installed on it by default. I'll see if the issue can be reproduced with it.

It seems the issue was fixed by https://github.com/narzoul/DDrawCompat/commit/8b393ea49876f8f7ee85182304bc265583b816c4.
More specifically, removing the o.pos.z = saturate(o.pos.z); line from VertexFixup.hlsl makes the stars disappear again.
The saturate function clamps its argument between 0 and 1, so the game must be setting the Z coordinates of stars outside of that range. Some quick debugging on top of v0.3.2 reveals that those coordinates are in the range of thousands. Whether the driver is actually supposed to clip such vertices or not, I'm not sure.
Here's the test build I was using, which isn't using the VertexFixup.hlsl yet, instead it fixes vertex coordinates via the CPU:
ddraw.zip (diff.txt compared to v0.3.2)
I added a quick fix into it, which clamps Z > 1 to 1, and also logs the coordinates of all problematic vertices in the log file. So don't look at the stars for too long, or the log file will be very big.

I added a quick fix into it, which clamps Z > 1 to 1, and also logs the coordinates of all problematic vertices in the log file. So don't look at the stars for too long, or the log file will be very big.

Yea its EXTREMELY big integers. Log attached
DDrawCompat-thief2.zip

I have the GOG version of the game, not sure what patches are installed on it by default. I'll see if the issue can be reproduced with it.

It installs at least T2Fix, although to my knowledge there is a Direct3D6 renderer still available inside.
Another query: how did you fix the gamma? Even T2Fix (the GOG patch) couldn't do it. 0.3.2 works, 0.2.1 can't run the game.

I don't think there is any fix for gamma in v0.3.2. Maybe exclusive fullscreen mode fixes it.
In later versions, when FullscreenMode is set to borderless, gamma ramps are emulated with shaders.

I don't think there is any fix for gamma in v0.3.2. Maybe exclusive fullscreen mode fixes it.

You are right, its the exclusive fullscreen. Thanks

Hi, thanks for the extremely interesting news.
I'd like to replicate your stars fix in DxWnd. May I ask you what D3D operation is actually responsible for the problem? I tried to trim all z coordinates to the 1.0 maximum value for all vertex in the IDirect3DDevice3::DrawPrimitive method, but with no success, the stars are still missing. Maybe I should hook some other method?

I'm not sure, because I'm hooking at the driver level, which handles all of them. Back when I tried to hook the various DrawPrimitive methods in the runtime, I found it to be too difficult and unpredictable, because the runtime uses some dynamic vtable and sometimes replaces some of these methods (and maybe others) with different method pointers, so it's hard to hook all variants. I gave up and settled on driver hooking instead.
Also, are you calculating the stride between vertices correctly? It depends on the FVF flags passed to DrawPrimitive. I guess if the first vertex has Z > 1, then you know you are in the right place, but the stride may still be incorrect for finding the rest of the vertices in the array.

Well, that fixing at the application level is not possible is arguable, but I don't want to miss a precious chance to learn how to put my hands on the 3D vertices at the driver level, a solution, I agree with you, enormously more flexible and valid for any DirectX 3D access, so I'm going to roll up my sleeves and try it.
So far I reached and hooked the pfnDrawPrimitive driver callback with a tracing proxy and it works, but the pfnDrawPrimitive seems to get some information about the drawing activity (like the number and type of elements) but it doesn't have access to the vertices data.
I was trying to get inspired (a.k.a copying) by your DDrawCompat code, but in this part the architecture is so different that the operation is not easy.
Could you help me clarifying some issues, that is the following:

what type of elements have to be fixed? (i.e. D3DPT_TRIANGLELIST, D3DPT_TRIANGLESTRIP, or D3DPT_TRIANGLEFAN or all three of them)
which data has to be trimmed? (the z coordinates)
what other driver callback should I hook to reach th data to be trimmed?

If you follow the DxWnd thread on SourceForge about this issue you'll see that one difficulty comes from the fact that you can't be sure about the effects of the code changes because the problem sometimes seems to vanish for completely different reasons.

what type of elements have to be fixed? (i.e. D3DPT_TRIANGLELIST, D3DPT_TRIANGLESTRIP, or D3DPT_TRIANGLEFAN or all three of them)

Probably all of them, and not even just triangles.

which data has to be trimmed? (the z coordinates)

Yes, Z > 1 -> Z = 1. DDrawCompat also does Z < 0 -> Z = 0, but I don't remember if any games need that.
Then there are also various RHW fixes which you may already be aware of, but it's an unrelated issue. Some drivers don't render RHW=0 and RHW=infinite, so I replace them with RHW=1.

what other driver callback should I hook to reach th data to be trimmed?

This parts gets a bit complicated because of different vertex sources and FVF, so here's the breakdown.
Driver functions referenced below are documented at:
https://learn.microsoft.com/en-us/windows-hardware/drivers/ddi/d3dumddi/ns-d3dumddi-_d3dddi_devicefuncs
Vertex sources
The driver supports both straight pointers to user-mode memory (e.g. IDirect3DDevice7::DrawPrimitive) and vertex buffers (e.g. IDirect3DDevice7::DrawPrimitiveVB).
User-mode pointers are received by pfnSetStreamSourceUm in the pUMBuffer parameter. This is the same pointer which is passed in lpvVertices of IDirect3DDevice7::DrawPrimitive.
Additionally, pData->Stride is the size of a vertex in the memory (determined based on the FVF code passed to DrawPrimitve, so you don't have to calculate it yourself).
You can ignore pData->Stream, because for transformed vertices it should always be 0.
For vertex buffers (not used by Thief 2), the stream source is set by pfnSetStreamSource. pData->Stream and pData->Stride can be treated as above.
pData->hVertexBuffer is a handle to the vertex buffer resource, created by either pfnCreateResource or pfnCreateResource2.
pData->Offset can be ignored, it should always be 0 for DX7.
DDrawCompat only ever supported CPU editing of system memory vertex buffer data, so that's the case I'll focus on. To get a pointer to the vertex data for a system memory vertex buffer, you need to track its creation via pfnCreateResource(2), and free the associated tracking data when the buffer is destroyed via pfnDestroyResource.
For pfnCreateResource(2), pData->Format = D3DDDIFMT_VERTEXDATA and pData->Flags.VertexBuffer = 1 for vertex buffers, and pData->Pool = D3DDDIPOOL_SYSTEMMEM to indicate it's in system memory. In this case, pData->pSurfList[0].pSysMem points to the vertex data. You'll need to save that pointer and associate it with the handle returned in pData->hResource, which matches the hVertexBuffer handle passed to pfnSetStreamSource.
The current vertex source can alternate between user-mode memory and vertex buffers, depending on which driver function was called last with valid arguments, so you need to track this information.
Flexible Vertex Format
I assume you are already familiar with FVF codes. We need this to determine if the vertex source contains transformed or untransformed vertices, because we only want to edit transformed vertices.
The driver doesn't receive FVF codes directly, but a different representation of it, which was introduced by DX9 and described in detail here: https://learn.microsoft.com/en-us/windows/win32/direct3d9/vertex-declaration
Anyway, we don't need a complicated analysis here, but some data needs to be tracked again. The vertex declaration is created by pfnCreateVertexShaderDecl. pData->NumVertexElements contains the number of elements in the pVertexElements array. The vertex declaration uses transformed vertices if any of the elements in pVertexElements contains D3DDECLUSAGE_POSITIONT in the Usage field. Save this information and associate it with the handle returned in pData->ShaderHandle. It can be released in pfnDeleteVertexShaderDecl.
The currently active vertex declaration is selected by pfnSetVertexShaderDecl. We only want to edit vertex data if the currently selected handle (hShaderHandle) is associated with transformed vertices.
Drawing non-indexed primitives.
You already found that pfnDrawPrimitive handles non-indexed primitive drawing. There is also pfnDrawPrimitive2, but I don't think DX7 uses it, so DDrawCompat doesn't even hook it.
Now that you have the stream source and stride, the rest should be fairly straightforward in this case.
pData->VStart is the index of the first rendered vertex, so multiply it by stride to get a byte offset into the stream source.
The number of vertices rendered from this offset is not received directly, you have to calculate it from pData->PrimitiveType and pData->PrimitiveCount. See getVertexCount in D3dDdi\DrawPrimitive.cpp.
Drawing indexed primitives
This is a fair bit more complicated. The main function is pfnDrawIndexedPrimitive2. There is also pfnDrawIndexedPrimitive, but it's not used by DX7.
dwIndicesSize is always 2 for DX7, so pIndexBuffer is practically always an UINT16 array that contains the vertex indexes. The number of elements in the index buffer matches the number of vertices to be rendered, which can be calculated the same way as in the non-indexed case.
pData->BaseVertexOffset is similar to VStart in the non-indexed case, but it's expressed in bytes, so divide it by stride (it should always be divisible without remainder) to get the base vertex index. Beware that this can be negative. Then, to get the actual index of a vertex, add the index from the index buffer to the base vertex index. Multiply it by stride to get a byte offset from the start of the vertex source.
The pData->MinIndex and pData->NumVertices are supposed to indicate the range of indexes in the index buffer (MaxIndex would be MinIndex + NumVertices - 1), before applying the base vertex index. This latter part is not obvious from the driver docs, but it's explained better in the DX9 SDK:
https://learn.microsoft.com/en-us/windows/win32/api/d3d9/nf-d3d9-idirect3ddevice9-drawindexedprimitive
However, based on my old results, DX7 doesn't fill these fields properly for some reason, so you'll have to calculate them yourself by finding the smallest and largest index in the index buffer. See the DrawPrimitive::drawIndexed method. I found that actually populating these fields properly before calling pfnDrawIndexedPrimitive2 can increase performance on old Intel drivers, which otherwise probably ended up processing too many vertices because of the incorrect initial values in these fields.
Then, you have two options:

Edit all vertices given by MinIndex and NumVertices, which could also edit some vertices that are not actually rendered.
Edit only the vertices referenced by the index buffer, but then you could end up editing the same vertex multiple times, unless you keep track of which ones you already edited (which might not be worth it for very simple edits).

Hi again and thanks so much for the very detailed and complete analysis of this complex situation.
I started to gradually hook and implement some of your directions hoping in some luck, but it seems I had not enough. In any case, what I did so far is enough to point to some doubts that may need clarification. My test case is the very beginning of the Thief II starting mission where I log the operations and try to make some coarse change to the vertices that should be evident at
a first glance. Here are some steps:

I started hooking pfnCreateResource adding some simple logs, but I got none, I supposed that this callback is not used here.
So I hooked pfnCreateResource2 where I got more logged operations. Here I didn't know how to interpret your sentence that follows:

For pfnCreateResource(2), pData->Format = D3DDDIFMT_VERTEXDATA and pData->Flags.VertexBuffer = 1 for vertex buffers, and pData->Pool = D3DDDIPOOL_SYSTEMMEM to indicate it's in system memory
If I have to map the word "and" in a logical and condition I get no calls that match the end of all three conditions. So I put the conditions in logical or and traced which element triggered the condition. Here is my log:

dxwnd.CreateResource2: hDevice=0xa410c0 pResource=0xdedf8
dxwnd.CreateResource2: SurfCount=1 handle=0xdbd6728
> reason: VertexBuffer
dxwnd.CreateResource2: hDevice=0xa410c0 pResource=0xdee78
dxwnd.CreateResource2: SurfCount=1 handle=0xdbd6b68
> reason: VertexBuffer
dxwnd.CreateResource2: hDevice=0xa410c0 pResource=0xdee78
dxwnd.CreateResource2: SurfCount=1 handle=0xb80db78
> reason: VertexBuffer

This should mean that the game creates three vertex buffers but none of them is created with pData->Flags.VertexBuffer set to 1 or in system memory. So, are these three vertex buffers suitable for the vertices transformation? Maybe this is a dead end for this situation and I should look for the other cases?
I don't know, but I'll start to explore also the other cases.

I started hooking pfnCreateResource adding some simple logs, but I got none, I supposed that this callback is not used here.

It can be used by older systems, so you should hook both whenever possible. The docs say pfnCreateResource2 is "Supported starting with Windows 8", but it might also depend on the WDDM version supported by the driver.
More precisely, in d3dumddi.h, you can see how some parts of the D3DDDI_DEVICEFUNCS struct are conditionally defined depending on D3D_UMD_INTERFACE_VERSION, but since this struct is filled by the driver, it's the driver's D3D_UMD_INTERFACE_VERSION that matters. You should not try to hook past what the driver supports, since you might be reading uninitialized or inaccessible memory. In other words, the size of the D3DDDI_DEVICEFUNCS struct returned by the driver depends on its own D3D_UMD_INTERFACE_VERSION, even if your own build environment supports a newer version. The driver's interface version is returned in OpenAdapter's pOpenData->DriverVersion.

If I have to map the word "and" in a logical and condition I get no calls that match the end of all three conditions. So I put the conditions in logical or and traced which element triggered the condition.

I meant all 3 conditions with "and", yes. If Format is D3DDDIFMT_VERTEXDATA, then Flags.VertexBuffer must be set and vice versa. It would be invalid not to set both at the same time. In my logs, it looks correct. DDrawCompat actually only checks for the flag, not the format. Make sure you're not mixing up Flags with Flags2, although the latter doesn't even have a VertexBuffer member, but maybe you're checking the full DWORD value, and not the member variable called "VertexBuffer" in the bitfield struct?
The system memory pool is partially my bad. This game doesn't use vertex buffers at all, but Direct3D creates some implicit vertex buffers during CreateDevice, and sometimes copies the user-memory vertex data there, instead of using pfnSetStreamSourceUm directly. And then of course it uses pfnSetStreamSource with those implicitly created vertex buffers.
The trick is that the driver initially tries to create these implicit buffers in video memory, and only falls back to system memory if it fails. Years ago, I observed some performance issues with the video memory versions, so I added code to intentionally fail their creation, by returning with E_FAIL from pfnCreateResource(2), but the actual error code probably doesn't matter much. A simple way to detect these implicit buffers is that apart from Flags.VertexBuffer, also Flags.MightDrawFromLocked will be set. Only these implicit buffers should ever use the latter flag. So check for those two flags and if the Pool is not system memory, return with some error.
Here are the logs from DDrawCompat for the creation of the first implicit vertex buffer, both the blocked video memory case and then the system memory fallback (excluding some uninteresting inner logs):
2570 23:12:49.894   > IDirectDraw4Vtbl::CreateSurface(&06B68170, *{0x5,0,65536,0,0,0,0,0,null,{0x0,0x0,0,0x0,0x0,0x0,0x0},{0x804000,0x0,0x0,0x0},0}, *null, null)
2570 23:12:49.894     > _D3DDDI_DEVICEFUNCS::pfnCreateResource2(&0799F000, *{D3DDDIFMT_VERTEXDATA,D3DDDIPOOL_VIDEOMEMORY,0,0,[{65536,1,0,null,0,0}],1,0,0,0,{0,0},&07C1E4C0,0x2080000,1,0x0})
2570 23:12:49.894     < _D3DDDI_DEVICEFUNCS::pfnCreateResource2(&0799F000, *{D3DDDIFMT_VERTEXDATA,D3DDDIPOOL_VIDEOMEMORY,0,0,[{65536,1,0,null,0,0}],1,0,0,0,{0,0},&07C1E4C0,0x2080000,1,0x0}) = 80004005
2570 23:12:49.894   < IDirectDraw4Vtbl::CreateSurface(&06B68170, *{0x5,0,65536,0,0,0,0,0,null,{0x0,0x0,0,0x0,0x0,0x0,0x0},{0x804000,0x0,0x0,0x0},0}, *null, null) = 80004005
2570 23:12:49.894   > IDirectDraw4Vtbl::CreateSurface(&06B68170, *{0x5,0,65536,0,0,0,0,0,null,{0x0,0x0,0,0x0,0x0,0x0,0x0},{0x800800,0x0,0x0,0x0},0}, *null, null)
2570 23:12:49.894     > _D3DDDI_DEVICEFUNCS::pfnCreateResource2(&0799F000, *{D3DDDIFMT_VERTEXDATA,D3DDDIPOOL_SYSTEMMEM,0,0,[{65536,1,0,&07C497E0,65536,0}],1,0,0,0,{0,0},&07C1E060,0x2080000,1,0x0})
2570 23:12:49.894     < _D3DDDI_DEVICEFUNCS::pfnCreateResource2(&0799F000, *{D3DDDIFMT_VERTEXDATA,D3DDDIPOOL_SYSTEMMEM,0,0,[{65536,1,0,&07C497E0,65536,0}],1,0,0,0,{0,0},&07C06C50,0x2080000,1,0x0}) = 0
2570 23:12:49.894   < IDirectDraw4Vtbl::CreateSurface(&06B68170, *{0x5,0,65536,0,0,0,0,0,null,{0x0,0x0,0,0x0,0x0,0x0,0x0},{0x800800,0x0,0x0,0x0},0}, *&07C6E7C8, null) = 0

Hi again.
I slowly started to hook all necessary methods and added some logging. The part that inhibits vertices on video memory seems to work, the selected resources are now built in system memory. Then I added an association table, and here I'm going to have some troubles, maybe I got your instructions wrong.
The table is enumerating couple of handles and memory pointers. In Thief2 I get vertices buffers from pfnCreateResource2. Then the handle is matched with the pData->hVertexBuffer field in pfnSetStreamSource but now I don't know how to associate the collected information with the pfnDrawPrimitive operation. pfnDrawPrimitive doesn't get the vertex buffer handle (it only receives the device handle that never matches with the vertex handle) and doesn't receive the vertex buffer address, so it seems that I miss a necessary link ... please, help!!
P.s. I say something obvious, but the game is creating several vertex buffers, so I should find a way to pick the correct one ...
Now some more: in order to trim the z values to 1.0 maximum there should be no problems, the operation is quick and safe enough to be repeated endlessly with no big impact. But I'm starting to grow a wicked idea: what if DxWnd could increase all x, y coordinates by a given factor but just once? Could that be a sort of magnifier or zooming feature? Maybe saving the original values somewhere so that the zooming could be chosen at will. But of course I have to fix the z problem first, then maybe I'll try to develop this other idea.

You need to use whatever vertex source was last specified via pfnSetStreamSource(Um). If the last such call was with a vertex buffer (pfnSetStreamSource case), then use that vertex buffer later at the time of pfnDrawPrimitive. If it was a user-mode memory pointer (pfnSetStreamSourceUm case), then use that pointer for pfnDrawPrimitive. Most driver functions other than pfnDrawPrimitive just set up the device state for drawing, so the driver uses the last received state values (be it the vertex source, render state, texture stage state, etc.).
Yes, you can modify x, y values to achieve zooming, but I'm not sure how you'd ensure that the values don't change again in the original vertex source, since the source is in system memory that the CPU could modify any time, without calling any driver functions. So it's best not to cache the results and just reapply the modification always to some internal copy of the vertex data. This is what DDrawCompat does. Of course, then you have to change the vertex source (with pfnSetStreamSource(Um)) to your own internal, modified copy before calling the real pfnDrawPrimitive.

Thank you. I think I put in practice most of your information, surely enough to add a z trimming that has no big side effects on Thief2. Surely there will be some errors, but I hope to be able to fix them.
About the zooming feature, I added it as an experiment, but this brings big problems and the result doesn't seem so exciting, so I'll put that aside for the moment.
Just for curiosity, I wanted to complete the logic adding the wrapper for pfnDrawIndexedPrimitive by copying and adapting what I did in pfnDrawIndexedPrimitive2, but I couldn't find where the index buffer starts in this case, there are no memory pointers in the arguments!
Sadly, on my testbed the z trimming works but the stars (that is, the full set of the sky textures) are still not shown after a change in the Options > Video > Sky Detail OFF and ON again. But maybe this is another problem like a game bug. I'll ask for the results of tests on other computers where the stars never appeared in the sky.

Yes, pfnSetIndices(Um) sounds like the right functions. I never bothered with them because DX7 doesn't support index buffers. They were added in DX8.

I found a bug in the DxWnd emulation. The Stars are back in the sky, but on a Sky Detail High -> Low -> High transformation, they go missing.
This behaviour was also observed on Ivy Bridge Intel drivers (pre 2021 refresh drivers) without any wrapper used

Hello.
The zooming feature is starting to work: I had to copy the indices and copy back after the scaling and also add a boolean control array to avoid multiple scaling of the same vertices (if you are curious I can share the code), and now it is working quite nicely. The main purpose of this feature is to have a quick control about the completeness of the vertices handling: any distorted image surely means that some vertex has not been processed as it should, likely because it was ignored and skipped.
So, for instance, this is the case of some scene in Thief 2: all the scene must come from a 3D rendering with textures, but for some reason the walls of an archivolt are not scaled like the other elements (the iron fence, the lamps ...). I'm attaching here the screenshots of the original and scaled scene.
So, the idea is that probably the hooking of pnfDrawIndexedPrimitive2 and pfnDrawPrimitive doesn't process all the vertices. In my coding there is also an incomplete tracing hook of pfnDrawIndexedPrimitive and this callback is not referenced too. Do you have some idea about what could be missing?
In addition, I met other situations where the vertex processing was not intercepted by this current implementation, in particular a pure D3D8 game (I don't recall the name right now ...) that seemed to ignore all my callback hooking.

I found a bug in the DxWnd emulation. The Stars are back in the sky, but on a Sky Detail High -> Low -> High transformation, they go missing.

It's not a bug in DxWnd, it happens with DDrawCompat too. I think it's just a game engine bug. The stars reappear again when loading a save game or restarting the mission. Maybe the vertices need to be loaded when the level is loaded, and switching to low detail unloads them, but switching back to high doesn't reload them, or something like that.

So, for instance, this is the case of some scene in Thief 2: all the scene must come from a 3D rendering with textures, but for some reason the walls of an archivolt are not scaled like the other elements (the iron fence, the lamps ...).

I could reproduce the issue with DDrawCompat by applying the zoom only when the stride is 32. The gate and the lamps use a stride of 40 bytes. So I'd assume you're not keeping proper track of the stride somewhere, or not advancing to the next vertex by using the stride as the offset between vertices. Both are rendered with non-indexed vertices, so otherwise it seems to be a relatively simple case.

In addition, I met other situations where the vertex processing was not intercepted by this current implementation, in particular a pure D3D8 game (I don't recall the name right now ...) that seemed to ignore all my callback hooking.

I don't know, I never hooked D3D8, but I'd assume it to be similar to the DX7 case, except maybe it also uses the other ones of the pfnDraw(Indexed)Primitive function pairs.

A short update to let you know just two things:

after some struggle, it seems that now the vertex scaling is working pretty well, though quite useless in most cases.
despite the "software traps" placed in the code, D3D8 seems immune to this trick, none of the callbacks mentioned above is ever triggered. Could it be that there other callbacks, or maybe they should be hooked in some different way? Just for curiosity ...

despite the "software traps" placed in the code, D3D8 seems immune to this trick, none of the callbacks mentioned above is ever triggered. Could it be that there other callbacks, or maybe they should be hooked in some different way? Just for curiosity ...

No idea, I never tried hooking D3D8.

I think I'm getting close to the end of this experiment. There are a few problems and questions, I don't ask for answers, just listing the issues.

I think I didn't understand fully the explanation about the Flexible Vertex Format. I suppose that once we find that the vertex source contains untransformed vertices we take note of it, so we should avoid making operations on the untransformed vertices on the following Draw*Primitive callbacks, . This would be fine for a Z value trimming, but it would be a problem for zooming because the zoomed picture would appear correct only if you scale all vertices.

as I said D3D8 seems to be avoiding this type of callbacks mechanism completely, so as other games like "Fable: the Lost Chapters".

another problem came from a game where I had high expectations, since the zooming would be quite useful in that case: "Project I.G.I.". Here I found a curious value of stride equal to 28 = 7 * sizeof(float). In all other cases I always got a 32 = 8 * sizeof(float) and the vertex addresses were successfully casted to a D3DVERTEX structure with the same size. In Project IGI the scaling makes weird effects that makes me thing that the X, Y offsets and sizes don't correspond exactly to those of a D3DVERTEX structure (among other oddities, the Y coordinate doesn't scale), but I don't know what data schema I should use in that case.


Consider adding a 'prepare' script to build when npm installing from git repo

With lambda-express in early development, features and fixes are getting added and need to be tested faster than new versions are being cut. As such, it would be handy to be able to
install lambda-express directly from the git repo. However, due to the build process and needing the dist files, this isn't currently possible using npm i <repo>. Fortunately, a prepare script was added in npm 5. This script is ran when installing from a git repo and can be used to build the needed dist files.
More info: https://blog.jim-nielsen.com/2018/installing-and-building-an-npm-package-from-github/#installing-and-building-packages-with-npm-from-github

Closed with #40


improved db.php to fix character encoding issues

Updated out of date functions and improved code standards and security. This was causing a bug in the way certain characters were being encoded after updating to WP 4.4.2

Wow, thanks! I I didn't realize there were so many changes to wp-db.php. Thinking more about this it seems like we should have our SSLed DB class match white-spacing of the core file seeing as it's mostly just a copy of it which should make future diffs cleaner when merging in changes from core:
https://github.com/xyu/heroku-wp/compare/update-db
The first two commits are just white-space updates to match core and the last, 4a1c36daedca4da70f75d8056d17ec08ca584b77, should be basically what you are proposing here does that look about right?

good idea, and yea looks good!
On Wed, 20 Apr 2016 at 20:53 Xiao Yu<EMAIL_ADDRESS>wrote:

Wow, thanks! I I didn't realize there were so many changes to wp-db.php.
Thinking more about this it seems like we should have our SSLed DB class
match white-spacing of the core file seeing as it's mostly just a copy of
it which should make future diffs cleaner when merging in changes from core:
https://github.com/xyu/heroku-wp/compare/update-db
The first two commits are just white-space updates to match core and the
last, 4a1c36d
https://github.com/xyu/heroku-wp/commit/4a1c36daedca4da70f75d8056d17ec08ca584b77,
should be basically what you are proposing here does that look about right?
—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
https://github.com/xyu/heroku-wp/pull/56#issuecomment-212534368

Closing this as the other branch has been merged thanks again @dmtintner


pdctl: fix subcommand option

What problem does this PR solve?
#1567 sets the detached mode as the default mode for pd-ctl, but introduces a problem which causes the flags in subcommand cannot be used, like --format for region key. Because it will parse the flags and return when meeting the unknown flag error.
What is changed and how it works?
This PR removes ParseFlags since it will be parsed when entering Execute.
Check List
Tests

Unit test

Need some unit tests?

@disksing PTAL.


Multiply two numbers

Write ruby code for multiplying two numbers.

Sir, can you please assign this issue to me

Hello @ShehrozIrfan. Kindly assign me this issue.

@Hegde2002
@sohampathak1234
Assigned.


[BEAM-8658] [BEAM-8781] Optionally set jar and artifact staging port …

…in FlinkUberJarJobServer
Pass around options to fix:

BEAM-8658: add an artifact_port option. We need this because we need to expose a static port to use for artifact staging on Kubernetes (AFAIK gRPC does not support in-process mode on Python).
BEAM-8781: respect the existing flink_job_server_jar option. I wanted this so I could download the released job server jar from Maven instead of having to build it locally. This allows me to override the default behavior for .dev, which is to fail: https://github.com/apache/beam/blob/0b415fd7e9dd5c80034ca237e08c3959ec78ffe3/sdks/python/apache_beam/utils/subprocess_server.py#L176-L181
We could consider changing that behavior (as indicated in the TODO), but it matters less if you can get around it if you really want to.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Choose reviewer(s) and mention them in a comment (R: @username).
[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Run Python PreCommit

For my reference, does anyone know if we publish nightly snapshots of the Flink job server jar anywhere?

I expanded this PR a bit to include the job and expansion ports as well. PTAL

Does anyone know if we publish nightly snapshots of the Flink job server jar anywhere?

https://repository.apache.org/content/repositories/snapshots/org/apache/beam/beam-runners-flink-1.9-job-server/2.18.0-SNAPSHOT/


Stripe.net version update

On January 1, 2017 Stripe will drop support for TLS 1.0 connections and SHA-1 certificates.
This article suggests that a minimum of Stripe.net 5.1.2 should be used. I believe Stripe.net 1.6.2 is used in this project and so will need upgrading for the January deadline.

Another one just did a pull request fixing PayPal and TLS 1.0 problems. Would you be up for fixing and send a pull request for Stripe? :)


XML export question.

A quick question, as I can't see it listed/asked anywhere, but when exporting a BPMN diagram from within Drawio using the XML export does it support the BPMN 2.0 XML notation. BPMN 2.0 specifications http://www.bpmn.org .
I believe it doesn't but can you confirm this?
Thx.

It does not.

It does not.

Hello. Does this still apply? It is still not possible to export a BPMN diagram from within Drawio using the XML export?

All updates to the subject are in this thread. It is not possible and we do not intend to implement it.


Dynamically loading markdown files

is it possible to use these function as if I were loading it from a variable instead of a specified path?
e.g.
// This
export let mdFile;

import { toc, html, attributes } from mdFile;

// Over This
import { toc, html, attributes } from '$lib/markdown/test.md';

I think it's impossible (and not the area of this plugin). vite offers "dynamic import" feature, that might be you're looking for: https://vitejs.dev/guide/features.html#dynamic-import


The discriminator should be trained more times ...

Hi @shaohua0116, thanks for your nice job!
However, I think one thing should be noticed, the discriminator should be trained more times than the generator.
After I change your code in the 'trainer.py' as follows:

the prediction accuracy improves quite a lot!
Hope you can notice~
thank you!

Please note that there's a trade-off between training the classifier faster and training the generator well so that the generated samples are able to help improve the generalization of the classifier.


Rekey harassment: client integration

This handles "phase 2" of rekey harassment as described in the confluence doc, with the exception of any OOBM handling.
In order to make this work, I implemented some of CORE-3100, but that isn't complete yet.
It is a bit wonky, but the tests pass.  Open to suggestions for improvements.
r? @maxtaco

This doesn't hit the real scoring endpoint.  I will implement that.

@maxtaco rebased for protocol changes, fixed some error issues, and this should hit the real scoring endpoint now.  PTAL.

had a few questions about behavior but otherwise looks great!

I need to make some changes to the protocol and the refresh calls for @chrisnojima.  I'd like to do those in a separate PR as this is becoming a long one...


Fuzz: limit input size

We know that fuzzParseExpr and fuzzParseMetricSelector make use of heavy
things like regexes, which take a fairly big amount of memory.
OSS-Fuzz does not offer a proper way to increase the memory [1], therefore
we limit the input size [2].
[1] https://google.github.io/oss-fuzz/faq/#how-do-you-handle-timeouts-and-ooms
[2] https://google.github.io/oss-fuzz/getting-started/new-project-guide/#input-size
Signed-off-by: Julien Pivotto<EMAIL_ADDRESS>

:+1:


Delete reminder

Implemented delete reminder command
Format: delete reminder INDEX
Close #48

Codecov Report
Base: 58.20% // Head: 57.08% // Decreases project coverage by -1.12% :warning:

Coverage data is based on head (322890a) compared to base (b890e09).
Patch coverage: 28.57% of modified lines in pull request are covered.

Additional details and impacted files
@@             Coverage Diff              @@
##             master      #63      +/-   ##
============================================
- Coverage     58.20%   57.08%   -1.13%
- Complexity      531      538       +7
============================================
  Files           110      115       +5
  Lines          2048     2104      +56
  Branches        239      246       +7
============================================
+ Hits           1192     1201       +9
- Misses          795      841      +46
- Partials         61       62       +1

Impacted Files
Coverage Δ

...main/java/seedu/address/commons/core/Messages.java
0.00% <ø> (ø)

.../seedu/address/logic/parser/AddressBookParser.java
68.18% <0.00%> (-3.25%)
:arrow_down:

...c/parser/reminder/DeleteReminderCommandParser.java
0.00% <0.00%> (ø)

src/main/java/seedu/address/model/AddressBook.java
71.42% <0.00%> (-2.35%)
:arrow_down:

src/main/java/seedu/address/model/Model.java
100.00% <ø> (ø)

...rc/main/java/seedu/address/model/ModelManager.java
63.52% <0.00%> (-0.33%)
:arrow_down:

...edu/address/model/reminder/UniqueReminderList.java
30.00% <0.00%> (-4.62%)
:arrow_down:

...reminder/exceptions/ReminderNotFoundException.java
0.00% <0.00%> (ø)

...logic/commands/reminder/DeleteReminderCommand.java
66.66% <66.66%> (ø)

src/main/java/seedu/address/ui/MainWindow.java
0.00% <0.00%> (ø)

... and 3 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.


Patch for pam tally2 is incorrect

Overall Review of Changes:
Updates regexp for 5.4.2 to match the line inserted by the patch.
It is still possible to define a value of ubtu20cis_pamtally2_login_opts that lacks onerr=fail and would insert a non-compliant fix -- but the default will now resolve the CIS requirement without generating new lines on every time the riole is applied.
Issue Fixes:
#80
Enhancements:
How has this been tested?:
Tested by applying repeatedly to a test server.

Please make sure you sign off on your commits. The DCO is failing because of that.

This has been added to devel branch from PR [#83] Please feel free to test it out and get back to us.


Fix indent issues in DG

Codecov Report

Merging #155 (8ad5cf7) into master (78d26cb) will not change coverage.
The diff coverage is n/a.

:exclamation: Current head 8ad5cf7 differs from pull request most recent head eb22b2f. Consider uploading reports for the commit eb22b2f to get more accurate results

@@            Coverage Diff            @@
##             master     #155   +/-   ##
=========================================
  Coverage     69.97%   69.97%
  Complexity      596      596
=========================================
  Files            94       94
  Lines          1905     1905
  Branches        245      245
=========================================
  Hits           1333     1333
  Misses          486      486
  Partials         86       86

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


No output while in a dos batch file

This is perhaps because the batch file is stopping the output, but ffmpeg-bar works perfect from the command line, but when running in a batch file, there is no output.  It is working perfectly (as in the files are created), but no output is shown during operation.
Sample one (this just concatenates video camera files)
`
@echo off
SETLOCAL ENABLEDELAYEDEXPANSION
mkdir n:__ftp_upload_watch_folder%1
for /D %%D in ("n:%1*") do (
for %%a in ("%%D.") do set currentfolder=%%~na
if exist "%%D*.mp4" (
echo Processing MP4 %%D ...
(for %%I in ("%%D*.mp4") do echo file '%%~nxI') > %%D\mylist.txt
)
if exist "%%D\*.mts" (
    echo Processing MTS %%D ...
    (for %%I in ("%%D\*.mts") do @echo file '%%~nxI') > %%D\mylist.txt
)

cd %%D
ffmpeg-bar -f concat -i "%%D\mylist.txt" -n -c copy "n:%1%1_!currentfolder:~0,2!_1.mp4"
echo !currentfolder! > "n:_ftp_upload_watch_folder%1%1!currentfolder:~0,2!_1_compress.title"
)
`
This is my second file which then compresses the newly minted concatenations.  Same thing, it works, but no progress-bar:
@echo off SETLOCAL ENABLEDELAYEDEXPANSION mkdir n:\__ftp_upload_watch_folder\%1 cd %1 (for %%i IN (*.mp4) DO ffmpeg-bar -hide_banner -n -i %%i -vf scale=1280:-2 -c:v libx264 -r 29.97 -crf 23 n:\__ftp_upload_watch_folder\%1\%%~ni_compress%%~xi )
Thanks!

Same issue when ffmpeg-bar is run from a python3 script via os.system().
This alternative works for py3:
subprocess.run(ffmpeg_command, shell=True)

You should fix your code markup above if you really want help, but that aside I find in linux bash scripts:
Any time the ffmpeg command is inside nested loops I need to redirect ffmpegs output or it can break the loop... obviously you  need appropriate syntax for windows.
Try slapping the windows-syntax equivalent of this at the very end of your command:
</dev/null
eg
ffmpeg .....-f mp4 -y "${exportName}" </dev/null


Need to address hashed password changes for batch invite

Related Issues
Fixes <link_to_github_issue>
Related PRs

branch
PR

service a PR
Link to PR

service b PR
Link to PR

Description
A few sentences describing the overall goals of the pull request's commits.
Must

[ ] Tests
[ ] Documentation (if applicable)

Only the import additions to descope/__init__.py are needed, thanks. Regarding the other changes, we support both a cleartext and prehashed password in the import so the field name should be password rather than hashed_password.


Feature Request : Connection Laplacian (to project DiffusionNet gradients in spectral basis)

Thank you for this super useful tool !
At present, we cannot have access to the connection Laplacian operator used for Heat Diffusion on tangent vector fields (defined on vertices).
It would be useful to be able to access it from the solver (with point cloud and with mesh if possible),
for instance L = solver.connection_laplacian()
The idea would be to use this laplacian to write gradients (defined at points) in spectral basis.
In this spirit, it could be useful to access gradient operators from within the solver too (since they have to be written in the local complex basis at each point which has to be the same as the one for the laplacian I suppose)
for instance G = solver.complex_gradient(). Alternatively, one could use for instance gradients defined in DiffusionNet but they would have to agree with the local basis of the connection Laplacian of the solver.
I hope this is enough information, thanks again for your huge help !

Hi Nick, thanks so much for the great tool.
It would be also useful to have an api to the point cloud connection laplacian, with the similar syntax L = solver.connection_laplacian() as in the case of mesh (which has been introduced already in the code base).
Do you think it's easy to introduce a projection operator, which projects (general) vectors defined on each vertex onto its tangent plane? The projected tangent vector component should be also represented in the local basis (used by the connection laplacian internally.) A possible api for this could be: vectors_tangent = solver.proj_tangent(vectors).
Many thanks in advance for your huge contribution to the community!


modules/auth/password/hash tests are slow

From https://github.com/go-gitea/gitea/actions/runs/4888003655/jobs/8725237392:
✓  modules/auth/password/hash (1m51.007s)

Almost two minutes for this seems far too slow.

scrypt spent 10x times than other algorithm.


Tags/commits aren't PGP signed

Should be following best-practice used by parent Bitcoin Core project.

Any progress on this, @mikehearn?  I wanted to install Bitcoin XT, but without a .sig for the OS X download (and without knowledge or instructions on how to build it myself), I am reluctant to install what is on the .dmg.

It's pointless to use a PGP signature for these downloads. They are served over SSL and you don't have a path to my PGP key through the web of trust anyway so you'd have to download it ... over SSL.

I suppose this is fine for now, but if Bitcoin XT becomes a more common client (which could happen with all the talk about 20 mb blocks), you should really look into this.
And a SSL connection doesn't prevent all threats - a PGP signature would just be another layer of optional security (of debatable security), which couldn't hurt.  Bitcoin Core downloads on the https Bitcoin.org website has a SHA256SUMS.asc to verify downloads - perhaps you could at least add SHA checksums for the downloads?

If we do the block size change, then I guess I'll team up with Gavin and he can sign things. His key is likely more widespread and trusted than mine.
Sure, I'll look at adding SHA256 sums to the next release.

@mikehearn, you could look into https://keybase.io to build trust in your key using your online accounts without getting into the web of trust.
If you sign up in the next few hours, the code "shit-yeah-facebook" will get you an account right away.  Otherwise you have to get an invite (it's still in alpha) or wait in the queue.

I could link my PGP key with github, but it'd boil down to an almost exact equivalence - it just makes github into a CA.

If you do decide to sign up for Keybase, you can also verify from other accounts (Reddit, Twitter, Coinbase, your website, etc.).  And as the message is PGP signed on all those sites, those websites are more of an intermediary for verifying your identity than a CA.
SHA256 sums sounds good for the next release.

@mikehearn step away from the keyboard....

@TheAwer there are SHA256 hashes of the downloads in the gitian release files:
https://github.com/bitcoinxt/gitian.sigs/blob/master/0.10.2A-linux/mike%40plan99.net/bitcoin-linux-0.10-build.assert
and that file is signed with my PGP key which you can get here:
https://vinumeris.com/mike-hearn.asc
Is that sufficient?

Yeah, that looks like an okay setup, @mikehearn (although it could certainly be more streamlined - like one file with all the hashes in it.  This should be enough for those who want to check, though.).  Only problem is that I can't find the hash of Bitcoin.XT.dmg.  It seems that you skipped this file for some reason or something?  The closest match would be the hash of bitcoin-osx-signed.dmg, but I'm not sure where that file is from.  From a quick check it appears that you do have the hashes of the rest of the files.
Did I miss this somehow?

Bitcoin.XT.dmg is that file renamed. I think the hashes should match unless the Finder twiddles stuff inside the DMG when renaming it.
Like I said, checking PGP signatures on these things doesn't tell you much, as the root of trust is still the PKI anyway, and the binaries on Win/Mac are signed as well. So I don't want to spend time on streamlining this right now. Data from bitcoin.org logs shows that ~nobody verifies PGP signatures. It's tantamount to security theatre.

My bad, I didn't actually check the hashes to see if it was the same.  It is.  Thanks for doing this - having this system is still better than nothing.  And I think that it's important to have security features like PGP sigs for those users who care enough to check this (I am one of those who check the PGP signatures).

Bitcoin is either business or show business. In both cases security theatre has value. And it keeps devs honest and is a path towards safer binary distribution. Doesn't have to be perfect from the beginning. OTOH, users should compile it themselves.

There are GPG signed gitian reports that you can compare against the downloads.


Refactored s3upload.py to reduce imports

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
Refactored some code in s3upload.py to reduce imports.

It's been a few months with no changes - I'm going to go ahead and close this.
Feel free to open one up if you get around to those changes!


Update contributing.md & Readme with additional info and section

I've added additional information in the contributions steps, such as one troubleshooting issue, additional commands to enable a smooth commit. As well as adding a new section to the readme & sharing my goals and ideas :)
This is my first PR on Open Source 🎉  Thanks in advance for your review! 🙏

One thing I actually realize is that git push origin main did not work for me, but doing git push origin Agathebadia:ab/contributions/my-first-os-contribution does work.
Not sure if this also needs to be updated in the contributing.md or if it's a local issue only I have.

Thank you, merged!


Rename flag 'placeholder' to 'receipt' and define a proto

Summary of changes:

Renamed flag placeholder to receipt https://github.com/apigee/registry/issues/276
Define a proto for ReceiptArtifact https://github.com/apigee/registry/issues/274

Thanks for the feedback!


Geoserver terria

Creating a hybrid Terria Catalogue

@Indiphile as discussed earlier, could you please make the Terria/Coastlines changes in one PR and the deletion of the crop masks in another PR in another branc? Thanks


Use common public key format in JWT user token for sshKey

Right now, our auth-server creates tokens with an sshKey attribute, which only reflects the keyValue column in the database... example:
{
sshKey: 'AAAAB...==',
sub: 'ansible-test',
iss: 'auth-server.dev',
role: 'user',
aud: 'api.dev',
iat:<PHONE_NUMBER>
}

It's very hard to remember the format of the sshKey attribute, since it is kinda "stringly typed" with certain expectations... without any validation in between. So it would be easier to use a value which is more common in SSH... the full public key string.
Expected output:
{
sshKey: 'ssh-rsa AAAAB...==',
...
}

Required changes:

[ ] Change token creation in auth-server / commons jwt module
[ ] Make sure the api/auth.js middleware uses this value correctly for looking up permissions (WHERE type = 'ssh-rsa' AND value = '...')
[ ] For local dev: Invalidate all apps using wrongly generated tokens

I agree, let's do this

Since I am already working on the auth middleware, I think it's best to just add this as well... I also think I already got all the changes needed to make this happen.

The proposed changes were introduced in #128 🎉


Run stress test for 24 hrs and verify results.

Need to run with and without agitation.

did not run with agitation for beta


Fix several issues with renaming in FileSystem dock

Fixes #82074
Fixes #82543
I tested all the cases I could think of, with Single Window Mode on/off & Split Mode on/off, and didn't spot any regressions.

Rebased to replace the focus check that was merged with #81007, similar to the others.
Checking if the tree has focus doesn't work when clicking outside the FileSystem dock, but checking the edited item always works.

It is good to see, that we came to the same conclusion of how to fix #82543.

Included an experimental fix for #19592.
I couldn't reproduce the issue on Windows 10 or Windows 11, so I included the macOS preprocessor check.
Can any macOS users test it?

Included an experimental fix for #19592. I couldn't reproduce the issue on Windows 10 or Windows 11, so I included the macOS preprocessor check. Can any macOS users test it?

CC @bruvzg

It is fixing it on macOS, on a case-insensitive file system. But it will overwrite file without any warnings on case-sensitive FS!

Probably require something more complex (work on both case-insensitive and case-sensitive fs on macOS, have not tested it on Windows).
https://github.com/godotengine/godot/pull/82957

Thanks for the review and quick PR!
I reverted the macOS change.

Included a fix for https://github.com/godotengine/godot/issues/82543#issuecomment-1751042987, which is a recent regression from https://github.com/godotengine/godot/pull/82045.
I'm not sure if it's the best solution though, I'm open to suggestions.

BTW now that you're familiar with that code @VedatGunel, you might be interested to look into #82774 (but if you do it's probably best left for a follow-up PR, as this one is fixing a regression while #82774 seems to be a pre-existing issue).

Thanks!


pull request for e2e test at 01/24/2017 00:26:15

:white_check_mark: Validation status: passed
For more details, please refer to the build report.
Note: If you changed an existing file name or deleted a file, broken links in other files to the deleted or renamed file are listed only in the full build report.


cmake: compiler: add warning for double-promotion

Add-Wdouble-promotion to the warning base. C implicit
promotion rules will want to make floats into doubles very easily. Given
that zephyr targets lower end microprocessors which do not have a
double precision floating point unit, this should be avoided or at least
known when it does happen.
This is to prevent something like #38651 from happening again
Signed-off-by: Ryan McClelland<EMAIL_ADDRESS>

Apparently the Automated checks are revealing there are now warnings being revealed :/

There are also warnings generated with this change when the CI runs their builds. Some of these are within the modules, such as STM32.
While I definitely can submit another pull request (or add another commit stacked below this one) that will fix the double promotion warnings within the zephyr repo... I'm not sure what do you about the ones in the vendor modules. I do believe that these should be fixed before merging in with this flag enabled by default.

The warning levels were inherited from the Linux kernel's Kbuild build system.
They have not proven to be particularly useful in Zephyr.

One could consider omitting the flag for native_posix.
double's are plenty fast outside of embedded.
If most failures are in native_posix code then omitting it for this platform could make it easier to get this in.

As I run through all the CI errors... There appears to be a lot of issues that are getting exposed (more than just the posix issues seen earlier)

Networking and Sensor drivers that were getting double promoted
ST HAL module has some issues with double promotion (mostly with the new STM32U hal they just added)
Just about every printf with floats which were always getting promoted to doubles (really no short term way around this other than to cast it :confused: )

For some modules, such as CMSIS-DSP, which use floats all over the place and have warnings in quite a few places. I'd rather not touch ARM's code, and just disable the -Wdouble-promotion flag for that module. It is not quite clear to me how disabling the flag for the module would be done. Would anyone have any insight?

zephyr_library_compile_options(-Wdouble-promotion)
in the CMakeLists.txt that creates the zephyr_library() which contains the sources.

Ahh.... I had to use zephyr_compile_options(-Wno-double-promotion) and that worked

You meant zephyr_library_compile_options, right? Otherwise you would disable it globally.

Opps.. yep, that's right. I used zephyr_library_compile_options(-Wno-double-promotion)

It might be best if this pull request is broken up. Gonna resubmit this to fix only the double promotions within zephyr itself, and the flags will have to come later.

unfortunately, there was an issue where I submitted that pull request with my main branch, and it won't let me rename it as a I draft another separate pull request


Fix project building from paths with spaces

Description
This little PR fixes a minor annoyance I faced when building the project - namely, a few files did not build successfully because of me having repository checked in a path with spaces.
Most files already had their paths enclosed in quotes, so fixing those few outstanding ones fixes building fully.

Thanks!


Fix for build script

Signed-off-by: Christian Lechner<EMAIL_ADDRESS>Description
Changed the build.sh script as proposed in issue  #177 (original issue #175):

added set -e to abort after first error
removed forward of tsc output to \dev\null

Issue reference
#177
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list:

[X] Code compiles correctly
[ ] Created/updated tests
[ ] Extended the documentation

I will merge this already for #177 but it only fixes it partially.

Thanks for merging, Yes well aware of that, but did not want to anticipate a solution as the fix can be done via npx or via installing tsc globally.


Added support for integers in hexadecimal notation

The added lines provide support for using integers in hexadecimal notation in
expressions. The code uses a regex to detect hexadecimal numbers.
Unit tests have been implemented to validate the implementation.

Thank you


[Refactor] Remove unused protobuf type CompressKind

compress_kind was set but never used.
What type of PR is this：

[ ] BugFix
[ ] Feature
[ ] Enhancement
[x] Refactor
[ ] UT
[ ] Doc
[ ] Tool

Which issues of this PR fixes ：

Fixes #
Problem Summary(Required) ：

Checklist:

[ ] I have added test cases for my bug fix or my new feature
[ ] I have added user document for my new feature or new function

[FE PR Coverage Check]
:heart_eyes: pass : 0 / 0 (0%)

[FE PR Coverage Check]
:heart_eyes: pass : 0 / 0 (0%)

run starrocks_admit_test


Display Wrong Place

Hi
I just setup your packages and browser shows me nearly 10 mil away. Also my Ar Drone wont take affect when i created a route to follow.

I'm no longer supporting this library as I don't use it any longer so I'm going to close this issue.


Wrong resolution dowloaded

Hi,
I'm trying to download a video in 480p resolution, but it ends up being 360p. In the preferences I've checked "Prefer videos with a resolution 480p ". Other similar videos behave the same way.
e.g. - https://www.youtube.com/watch?v=FHFU2GnPfm8
Am I doing something wrong?

You're not doing anything wrong, its most likely that YouTube didn't separate the video and audio for the 480p version so my program ignores it. Right now if the video and audio are not separated my program can't download it. Its just a technical issue, its not impossible to do it.
I don't have plans on supporting it because this kind of videos are rare and I don't have much time on my hands, sorry
You can create a pull request if you know how to code and I will happily merge it, and if you need help with how the code works I can help you.
Good luck!


[Build] [Windows] Build Error:  Warning treated as error

There is a build error due to Warning treated as Errors policy when building from Visual Studio in src\cpp\server\server_builder.cc line 273: conversion from 'size_t' to 'int', possible loss of data.
Fix sould be something like:
- int num_frequently_polled_cqs = sync_server_cqs->size();
+ size_t num_frequently_polled_cqs = sync_server_cqs->size();

Which version of the compiler and which platform?

Sure:

Visual Studio 2015 Community, version 14.0.25425.01 Update 3.
Target: Release, x64.
Win 7 x64 Enterprise


Filtering on supranational level/groups of countries?

It would be nice if it was possible to filter/select all the countries in the European Union and similar.
Possible and interesting things to filter on would be:

Unions (like the EU)
Continents (like Africa)
Alliances (like NATO)
Geographic selections (like Scandinavia)
Cultural regions (like the Balkans)

Some ideas:

Unions (like the EU): easy to query for since we have member of (P463) but hard to select which one's to select. All?
Continents (like Africa): easy  we have continent (P30)
Alliances (like NATO): same as unions
Geographic selections (like Scandinavia): possibly part of (P361) but there seems to be a lot of missing data

What about making something similar to topic pages but for supranational entities?

That makes a lot of sense!


add duphold depth filtering

Second step in closing #761
Will filter vcfs based on a del/dup depth added by the duphold.cwl in #763 and pass the other sv calls. Currently Manta has other calls with svtype of BND and INS while Smoove produces calls with BND and INV

I'll add the parentheses to be consistent. For the expression I followed the example provided in the duphold README https://github.com/brentp/duphold#duphold-uphold-your-dup-and-del-calls


Distributed training for CERN use case

Update the core API, if needed, to support distributed training for CERN use case (pytorch lightning and tensorflow)

Working now


mesos: no resources available to schedule container

Hi, I have a Mesos cluster consisting of 2 agent nodes:

I started 2 containers yesterday like the following:
docker run -d -c 1 -m 100 busybox sleep 999
docker run -d -c 1 -m 100 busybox sleep 999

and I can see they are completed now:

However when I try to schedule another container I'm getting this “no resources available” error:
➜  ~  docker run -d -c 1 -m 100 busybox sleep 999999
FATA[0005] Error response from daemon: no resources available to schedule container

I tried cleaning up some stopped containers but
docker rm -f `docker ps -aq`

is taking forever (been a few minutes, not returned yet, docker rm is getting stuck). I'm setting -c 1 -m 100 it seemed okay and the cluster has plenty of resources offered. Any ideas what's going wrong here?

Hi there, I think there might be some problems around resource accounting in the swarm mesos scheduler that we need to figure out. Another thing is that we're refactoring the scheduler so we don't use a full offer per task, but try to run multiple tasks on a offer when we can. So short answer is that once we have the refactoring merged hopefully this will be fixed too.

+1

This makes Swarm+Mesos completely unusable. What is even worse is that SWARM is sucking up all the resources from other frameworks too. @tnachen can you point me at the refactoring work you mention, this is a blocker for me right now and I'd like to follow along and at least provide some testing, if not patches.

@ahmetalpbalkan hey, sorry for the delay, we are aware of the issue. Can you try with https://github.com/docker/swarm/pull/1212 to see if it helps ? thanks.

This will not work with the current PR for now, haven't updated with the latest commits. Will update here as soon as the it is updated with a reasonable solution, it's more tricky than expected because:

One might remove a container not going through swarm or the Mesos cluster driver, so the resource-accounting must listen to the events cluster-wide and deal with adding/removing available resource chunks
The resource accounting might alter the existing scheduler for Mesos, this will require some changes here as well
I want to make sure that this will work for additional cluster drivers

Also the resource accounting happens in Swarm now, but we really should rely on docker/runC to do the resource reservation in an optimistic and atomic way instead of relying on our weird mapping of vcpu -> CpuShares. This works fine for Swarm at a higher level but we should have a more robust resource reservation mechanism at a lower level, especially for pluggable cluster drivers like Mesos.
Also it feels wrong to lock a Node (or any) object because two requests are trying to reserve resources at the same time. docker should return an error it one request cannot reserve a given amount of resource, and swarm should handle a retry mechanism at a higher level.

Not sure what you mean by relying on docker/runc for the resource reservation, but I think there is a bit of impedence mismatch here since if you rely only on docker for resource on the other hand your available resources is not given by docker but by Mesos, so you could get out of sync.
And about locking, If you don't lock than effectively you use compareAndInc/Dec for accurate accounting, but since you have multiple resources then I'm not really sure this makes your logic even clearer.

Sorry for the confusion, Mesos was a bad example for the whole second part. Because we rely on Mesos for resource offers and inform us of available resources.. But as the refactoring does not concern only Mesos I got lost in my thoughts...
Second part concerns mostly swarm and hypothetical cluster drivers relying on docker to do the resource accounting. So this was out of topic for this issue.

I meet a similar issue in my cluster (Swarm 0.5.0-dev + Mesos 0.25.0); when I run sleep 10000000, docker cli return "no resources available ", but it's running in the slave host & showing in Mesos GUI.

I confirm that this appears to work. At least on the tests I've done so far I'm not seeing the issue. Thank you @vieux


options for quote style single/double

Great extension ... one nice addition would be if we could either define the preferred quoting style or even better, the extension looks for a tslint/eslint.json to infer the rule defined.
Currently when scaffolding a construct I always have to fix up the quotes to satisfy the linter

I'd love to have back-ticks instead of single quotes too :)

A possible solution here: [How to access user settings for a snippets extension in VSCode].(https://stackoverflow.com/a/55140140/2443005)
Unfortunately, it'd be a big overhaul as it cannot be done in JSON.

This requires implementing a CompletionItemProvider and returning the snippet from that, rather than statically declaring it in a JSON

I was excited to find this extension, but it's not going to work out if I'm stuck with conflicting quote styles -- my linter won't be happy, and it's more work to fix the quotes than to just write (or copy/paste) the test skeleton by hand.
It looks like the project isn't being maintained anymore.  Is one of the forks under active development?

Been a long time, but I finally published an extension based off of this one that allows you to select your framework (Jasmine, Mocha or Jest) and your preferred quote Style.  It also fixes some of the other issues.
https://marketplace.visualstudio.com/items?itemName=JaredE.javascript-test-snippets


Session result individuals

Relates #34
Description:

This PR as part of EPIC SessionResults #31 .
IndividualResults show the result for each response in a single view.
Get session responses from backend using /session/responses/:sessionId/:sessionType
This will get abject with {answers & questions}
the front end will join each answer with a related question using question ID, and get a list of responses array.

Codecov Report

Merging #77 into master will decrease coverage by 1.64%.
The diff coverage is 36.47%.

@@            Coverage Diff             @@
##           master      #77      +/-   ##
==========================================
- Coverage   42.67%   41.02%   -1.65%
==========================================
  Files          67       71       +4
  Lines         689      758      +69
  Branches       58       67       +9
==========================================
+ Hits          294      311      +17
- Misses        365      408      +43
- Partials       30       39       +9

Impacted Files
Coverage Δ

server/database/db_connection.js
100% <ø> (ø)
:arrow_up:

...ient/src/Components/Layouts/SessionResult/index.js
0% <ø> (ø)
:arrow_up:

client/src/Components/Questions.js
0% <0%> (ø)
:arrow_up:

server/controllers/session.js
0% <0%> (ø)
:arrow_up:

server/controllers/get_session_responses.js
0% <0%> (ø)

server/database/queries/get_session_answers.js
100% <100%> (ø)

...mponents/Layouts/SessionResult/StyledComponents.js
84.78% <100%> (+4.22%)
:arrow_up:

server/database/dummy_data_build.js
100% <100%> (ø)
:arrow_up:

...onResult/IndividualResults/join_question_answer.js
12.5% <12.5%> (ø)

...ionResult/IndividualResults/get_responses_array.js
16.66% <16.66%> (ø)

... and 8 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 693a24b...0c064fe. Read the comment docs.

fixed


Migrate to copier profile and centralize workflows

What does this PR do?
https://github.com/salt-extensions/saltext-azurerm/pull/42
Commits signed with GPG?
Yes
Please review Salt's Contributing Guide for best practices.
See GitHub's page on GPG signing for more information about signing commits with GPG.

Welcome to Codecov :tada:
Once merged to your default branch, Codecov will compare your coverage reports and display the results in this comment.
Thanks for integrating Codecov - We've got you covered :open_umbrella:


Add Granian to ASGI and WSGI deployment guide

Granian is a Rust HTTP server for Python applications which supports both ASGI and WSGI application protocols.
It isn't as well known in the Django community as an option - hopefully this will help that.

@gi0baro You may want to consider adding it to djangopackages.org: https://djangopackages.org/packages/add/

https://djangopackages.org/packages/add/

Thank you for the hint! Just added

I'm personally interested/excited about Granian, but I don't think we should be adding it to the docs.
We're not in the business of listing every option.

That make sense.
I think it would still be valuable for people to have some documentation about specific features some servers can add to the game, for example:

Hypercorn is the only existing server out there providing HTTP/3
Granian is the only existing WSGI server that actually speaks HTTP/2 (even if it keeps saying 1.1 in WSGI environ as WSGI doesn't really expect it)

So maybe we could try rewrite the entire deployment chapter? Sure, it sounds huge, but again I think we can still deliver more valuable docs to the users in this regard.

Hey @gi0baro — TBH I still think that's a load more detail than we should be going into in the Django docs themselves. (There may be more suitable venues for such info, useful as is it when you're digging in more.)
(If we wrote afresh, I'd likely say just mention Gunicorn for WSGI and then Daphne or Uvicorn maybe for ASGI, and stop there.)

TBH I still think that's a load more detail than we should be going into in the Django docs themselves. (There may be more suitable venues for such info, useful as is it when you're digging in more.)

@carltongibson again, it makes sense. I get your point.
Actually, feel free to close this PR and eventually point me to more preferable venues :)

point me to more preferable venues :)

Maybe turn this into a blog post and share on the Django forum "Show and Tell" section (https://forum.djangoproject.com/c/projects/11) and re-post in the Django discord? :+1:

Maybe turn this into a blog post and share on the Django forum "Show and Tell" section (https://forum.djangoproject.com/c/projects/11) and re-post in the Django discord? 👍

Sure thing! Thank you all ❤

@gi0baro Thanks :+1: Closing per discussion.


[pyupgrade] Remove keep-runtime-typing setting

Summary
This was implemented "by accident" before realizing that it's exactly equivalent to turning off two of our rules. (Pyupgrade doesn't have a generic mechanism for turning off individual rules, and so has dedicated flags for them.)

I don't think the two mechanisms are exactly equivalent.  The benefit of having a stand-alone setting for it is that the rules will take effect immediately upon bumping the minimum supported Python version, whereas now the user will have to remember to un-exclude the rules they've been forced to exclude for runtime typing compatibility, and the burden of tracking compatibility per Python version will have fallen on them.

@layday - I understand that perspective, but these rules already take the minimum compatible version into account (and/or the required presence of from __future__ import annotations), so I don't believe users should have to consider their own minimum supported version at all when enabling or disabling them, or toggling the keep-runtime-typing  settings. There was more discussion of this here (https://github.com/charliermarsh/ruff/issues/3174), long ago -- but, e.g., the only reason to turn these rules off or set keep-runtime-typing = true would be if you had some application behavior that depended on the availability of runtime typing, and in that case, it seems like changing the Python version would be irrelevant to whether that decision changed.
As implemented in Ruff, this setting was exactly equivalent to enabling and disabling these rules. But I'm happy to continue discussing or reconsider if you have an example of where this would cause problems.

[...] would be if you had some application behavior that depended on the availability of runtime typing, and in that case, it seems like changing the Python version would be irrelevant to whether that decision changed.

Ah, but the assumption is that the application is able to handle these new typing constructs at runtime, the only limitation being the lowest Python version that it supports, i.e. that evaluating stringified annotations does not raise.

As implemented in Ruff, this setting was exactly equivalent to enabling and disabling these rules.

To clarify, bumping target-version to 3.10 would not have enabled either of these rules?

To clarify, bumping target-version to 3.10 would not have enabled either of these rules?

That's correct. It used to only affect the pre-3.10 branch, but I changed it in #2449. I may have misunderstood the motivation for the flag, and it's possible that our enforcement of it was incorrect, though I still don't fully understand why it exists. It seems like the example of a case in which users want to use this flag is on pre-3.9 code like:
from __future__ import annotations
import pydantic

class User(pydantic.BaseModel):
    first_name: str | None = None
    last_name: str | None = None

But in this case, why is __future__ annotations enabled? Is it not unsafe to use __future__ annotations at all if you're relying on runtime typing in this way?

One might also use from __future__ import annotations to make recursive or cyclic type hints nicer, eg:
from __future__ import annotations

import pydantic

class User(pydantic.BaseModel):
    friends: list[User]

(here's a non-contrived example)

This may be fairly minor, but if there are future syntax incompatible typing PEPs introduced other than PEP 585 and PEP 604, one other difference is that users (as opposed to ruff) will need to be responsible for updating their configs when making ruff updates in order to keep runtime typing.


Igonchar/tdi 44235 2 release script (#1676)

What is the new behavior?
https://jira.talendforge.org/browse/TDI-44235
Please check if the PR fulfills these requirements

[ ] The commit message follows Talend standard
[ ] Tests for the changes have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features) ?
[ ] The code coverage on new code >75%
[ ] The new code does not introduce new technical issues (sonar / eslint)

What kind of change does this PR introduce?

[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Build / CI related changes
[ ] Other... Please describe:

Does this PR introduce a breaking change?

[ ] Yes
[ ] No

If this PR contains a breaking change, please describe the impact and migration path for existing applications: ...
Other information:

Pull request test result: failure :x: