Project Oracle August 2016

https://github.com/Karrandalf/Dota2-TheCore-Config-Engine

@Chillrox, if you see this can you confirm that your in game hero builds are working again. Removing the custom mini-map seems to have fixed the problem for me.

@Karrandalf Hello! SOrry I just saw your message, problem is still up :( http://puu.sh/qCxYS/5e3aa6b5d7.jpg

@Chillrox , so try the updated version of Project Oracle on a fresh install of dota. The dark minimap background was causing the issue for me and I have since removed it from the project.
Alternatively, you could try deleting the gameinfo.gi file and letting the game generate a new one.

@Chillrox , If you do decide to delete the "gameinfo.gi" file instead of reinstalling, it is located in this folder
steam\steamapps\common\dota 2 beta\game\dota
You can also delete the entire folder "ProjectOracleMinimap"
steam\steamapps\common\dota 2 beta\game\ProjectOracleMinimap

@Chillrox
Oddly enough, I just figured out what was wrong with minimap folder.
Dota 2 was creating a folder called "guides" inside of "ProjectOracleMinimap" when it should be in "dota".
By changing the name of this folder to "dota_ProjectOracleMinimap" the "guides" stays inside of the "dota" folder.
I have just added the minimap replacer back to the repository. You will need to install this updated version to a clean copy of dota 2. To reinstall dota, just right click it in your steam library and click delete local content. Then re-download the game install Project Oracle.

@Chillrox, after you reinstall dota 2, make sure you load a bot match at least once before you install Project Oracle or else the in-game guides will break again.

I made the DarkMinimap mod optional now with instructions to enable it in the readme. After more extensive testing it seems the best way to make the in-game builds work with mods for dota is to use a symbolic link. I have included a batch file that will automatically create this link when run. Installing mods this way ensures that they will not be overwritten everytime dota updates.
@Chillrox, Sorry that fix yesterday did not actually fix the hero builds after I tested it on a fresh install. The fix I have in the repository now should fix the in-game builds. I have tested it on a fresh install on two different systems.

@Karrandalf I congratulate you! It's officially fixed :D

@Chillrox , ty for your dedicated testing. Let me know if you find any other bugs.


Define BLAS acronym in README

BLAS is a fairly common acronym in other parts of computing (ie. raytracing for bottom level acceleration structure.)
I saw an article about the project on Phoronix and has no idea as to what this project was about as it expected me to know what BLAS meant without ever defining it.
This PR makes it more explicit in the README about what the library does.

Hm, we have this information in the FAQ, but I was not aware that the acronym has been appropriated for other fields lately.
Generally the fundamental documentation is a bit scarce here as OpenBLAS sees itself as the continuation of GotoBLAS which was a novel concept to speed up BLAS, all under the assumption that "you already know what BLAS is" and are familiar with THE reference implementation at netlib.org and its accompanying documentation :smiley:

It is initialism for particular API, not referring to any Basic set of linear subroutines, like sandard C math libraries that are able to calculate simple linear projection.


Add virtual machine manager

Functionality extended to support Virtual Machine Manager API.
New methods include:
List all Virtual Guest Names
List Virtual Guest Statuses
Power on a Guest
Power off a Guest
Shutdown a Guest

Hi @realtek2017
I like this idea, so let's work on it to bring it on the road 😉
Overall please compare with the implementation of surveillance_station - it implements the generic class SynoSurveillanceStation which more or less represents the app itself and the class SynoCamera which represents the single camera instance with its properties like the name, id, model,...
Last but not least ... tests are also needed 😉

Hi @realtek2017 I like this idea, so let's work on it to bring it on the road wink
Overall please compare with the implementation of surveillance_station - it implements the generic class SynoSurveillanceStation which more or less represents the app itself and the class SynoCamera which represents the single camera instance with its properties like the name, id, model,... Last but not least ... tests are also needed wink
Btw I'm missing the reboot method wink

Thanks @mib1185 for all your comments, I appreciate your feedback.
My reason for adding this was driven from being able to use this eventually on the homeassistant integration :)
I will make the changes as requested, had some issues with getting poetry working so I will try and fix that for the tests.

My reason for adding this was driven from being able to use this eventually on the homeassistant integration :)

was hoping, that this is the reason behind 🤞

I will make the changes as requested, had some issues with getting poetry working so I will try and fix that for the tests.

If you are using VSCode and have Docker installed on your system, you can use the devcontainer in this project 😉

This will be implemented with #363


What is the need for nacl_helper_bootstrap on Linux?

What is the need for nacl_helper_bootstrap on Linux?
On my end the game runs without it.

I noticed it because FreeBSD can run the Linux binary on Linuxulator without the bootstrap, it also runs with the bootstrap. So I wondered if that was true on Linux too, it looks like it is.

@Kangz would you know what the help is for?

The source is here. It seems to be a custom dynamic loader that loads the NaCl ELF to set up part of the sandbox but I'm not sure how it is used, or what it does exactly.

There's a test that sheds some light on what it's supposed to do. Apparently the reserved_at_zero part is only expected to do anything on non-amd64 Linux.
One of the duties is to forbid memory mapping of some regions. Forbidding code to be mapped at certain regions is part of the secure sandboxing architecture. So you might not see something break but disabling it could let programs escape the sandbox.


Lambda expressions composing stopped working

Hi again. I think this got broken after .net core and/or ef core update to 2.0.6.
So I have this code in controller:
var u = await _dbContext.Users.Where(new UserBasic(4526660).ToExpression()).ToListAsync();
Where UserBasic is specification class. See attached file "Specification pattern". When I run this I get the error:
ArgumentException: Property 'Int32 UserId' is not defined for type 'GLXY2.Data.Entities.UserGroup.User'
Parameter name: property
System.Linq.Expressions.Expression.Property(Expression expression, PropertyInfo property)

// Here framework code logs //

GLXY2.Patterns.SpecificationPattern.CombineExpressions(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right, bool and) in SpecificationPattern.cs
+
            exprBody = (BinaryExpression)new ParameterReplacer(paramExpr).Visit(exprBody);
GLXY2.Patterns.SpecificationPattern.CombineAndExpressions(Expression<Func<T, bool>> left, Expression<Func<T, bool>> right) in SpecificationPattern.cs
+
            return CombineExpressions(left, right, true);
GLXY2.Models.Specifications.UserGroup.UserBasic.ToExpression() in UserBasic.cs
+
            return CombineAndExpressions(expression, expression);
GLXY2.Controllers.HomeController+<Index>d__7.MoveNext() in HomeController.cs
+
            var u = await _dbContext.Users.Where(new UserBasic(4526660).ToExpression()).ToListAsync();

// Here framework code logs //

So I'v tried many ways to debug this and only problem is with navigation property which works ok when in one expression and get lost when gets combined. Before I had something else with this thing and it's kind of disaster. I just can't trust what I code. Today it works and tomorrow it may not. Can you help me?
Further technical details
EF Core version: latest stable version 2.0.3?
Database Provider: Microsoft.EntityFrameworkCore.SqlServer
Operating system: Windows 10
IDE: Visual Studio 2017 Community latest stable version

When I changed code of CombineExpressions to this it became working again. I don't know anything about it, but why ParameterReplacer that worked well before, and which is well known on the web doesn't anymore? And am I safe with this new code? What is the parameter and can be there more than one while I combine only expressions of same T? And I don't know is this a bug in EF or C# language.

Your ParameterReplacer is incorrect. That is the reason it does not work.
In the ParameterReplacer whenever you are visiting a parameter node you are replacing it with _parameter which is of type user.
The expression you are calling parameter replacer contains another lambda inside. like this one u.MyIgnoredUsers.Where(ign => ign.UserId == _currentUserId).Count() == 0
There is lambda expression inside the Where which contains own parameter ign. By blinding replacing it with _parameter (of type User) in parameter replacer, you eventually caused a member expression of u.UserId which does not exist hence you get exception that UserId is not defined on User. Essentially the ExpressionTree is getting corrupted by ParameterReplacer.
To correctly replace one expression with another expression in ExpressionTree, you should utilize both the nodes.
You would need some code like this:
https://github.com/re-motion/Relinq/blob/76e03ac938a5581b60d67b3032fb7908bf929177/Core/Parsing/ExpressionVisitors/ReplacingExpressionVisitor.cs#L42-L57
In simple step, whenever you encounter old expression, return the replacement. You can figure out old & new parameter in CombineExpressions easily using left/right.Parameters.

@tuespetre - Sorry for posting same answer. I had stale tab open for few hours and i did not see it was already answered till I posted comment.


New version doesn't work

I run main.py as before and get the error:
Traceback (most recent call last):
  File "/Mi-Fit-and-Zepp-workout-exporter/main.py", line 8, in <module>
    from src.exporters.geopandas_exporter import GeoPandasExporter
  File "/Mi-Fit-and-Zepp-workout-exporter/src/exporters/geopandas_exporter.py", line 7, in <module>
    import geopandas as gpd
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/__init__.py", line 1, in <module>
    from geopandas._config import options  # noqa
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_config.py", line 109, in <module>
    default_value=_default_use_pygeos(),
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_config.py", line 95, in _default_use_pygeos
    import geopandas._compat as compat
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_compat.py", line 9, in <module>
    import pyproj
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/__init__.py", line 49, in <module>
    import pyproj.network
  File "/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/network.py", line 10, in <module>
    from pyproj._network import (  # noqa: F401 pylint: disable=unused-import
ImportError: dlopen(/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/_network.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_proj_context_is_network_enabled'

My pip list:
Package            Version
------------------ ---------
attrs              21.4.0
black              22.3.0
certifi            2022.6.15
charset-normalizer 2.0.12
click              8.1.3
click-plugins      1.1.1
cligj              0.7.2
Fiona              1.8.21
geopandas          0.11.0
gpxpy              1.5.0
idna               3.3
isort              5.10.1
munch              2.5.0
mypy-extensions    0.4.3
nodeenv            1.7.0
numpy              1.23.1
packaging          21.3
pandas             1.4.3
pathspec           0.9.0
pip                22.1.2
platformdirs       2.5.2
pydantic           1.9.1
pyparsing          3.0.9
pyproj             3.3.1
pyright            1.1.251
python-dateutil    2.8.2
pytz               2022.1
requests           2.27.1
setuptools         57.4.0
Shapely            1.8.2
six                1.16.0
tomli              2.0.1
typing_extensions  4.3.0
urllib3            1.26.10

Could you please try to remove/reinstall the pyproj package?

Reinstalled, doesn't work.
I also tried installing pyproj==3.3.0 and that doesn't work either

I don't think the new changes caused this issue, because according to the stack trace, it's coming from import geopandas as gpd, which has been already there before.

Which Python version do you use?
Which OS?
Do you have the same issue when you create an empty .py file containing only import geopandas as gpd?

I don't think the new changes caused this issue, because according to the stack trace, it's coming from import geopandas as gpd, which has been already there before.

Do you have the same issue when you create an empty .py file containing only import geopandas as gpd?

i created a new file test_import.py
posted the code:
import geopandas as gpd

print('hello')

Got this answer:
Traceback (most recent call last):
  File "Mi-Fit-and-Zepp-workout-exporter/test_import.py", line 1, in <module>
    import geopandas as gpd
  File "Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/__init__.py", line 1, in <module>
    from geopandas._config import options # noqa
  File "Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_config.py", line 109, in <module>
    default_value=_default_use_pygeos(),
  File "Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_config.py", line 95, in _default_use_pygeos
    import geopandas._compat as compat
  File "Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/geopandas/_compat.py", line 9, in <module>
    import pyproj
  File "Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/__init__.py", line 49, in <module>
    import pyproj.network
  File "Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/network.py", line 10, in <module>
    from pyproj._network import ( # noqa: F401 pylint: disable=unused-import
ImportError: dlopen(GitHub/fork/Mi-Fit-and-Zepp-workout-exporter/venv/lib/python3.9/site-packages/pyproj/_network.cpython-39-darwin .so, 0x0002): symbol not found in flat namespace '_proj_context_is_network_enabled'

Rewrote the code:
print('hello')

and got the answer:
hello

Which Python version do you use?

Python 3.9.7

Which OS?

macOS 12.4

It looks like it's a problem related to your Python environment instead of the project itself. I'm also using macOS 12.4, the only difference is the Python version - I'm using Python 3.9.13. I don't think it helps, but you could try updating it.
You may also try to recreate the venv from scratch.


How do I set my preferred backend?

I want to set DirectX 11 like backend, my Video Card support Vulkan, DirectX, OpenGL, but I want to set DirectX

You can check VeldridStartup.CreateGraphicsDevice method in the Veldrid.StartupUtilities package. There is a method override which enables you to select a preferred graphics backend.


[SPARK-39521][INFRA] Separate scheduled jobs to each workflow

What changes were proposed in this pull request?
This PR proposes to separate workflows for each scheduled jobs. After this PR, each scheduled build can be easily searched at https://github.com/apache/spark/actions. For example, as below:

In addition, this PR switches ANSI build to a scheduled build too because it's too expensive to run it for each commit.
Why are the changes needed?
Currently it is very inconvenient to navigate scheduled jobs now. We should use https://github.com/apache/spark/actions/workflows/build_and_test.yml?query=event%3Aschedule link and manually search one by one.
Does this PR introduce any user-facing change?
No, dev-only.
How was this patch tested?
Tested in my fork (https://github.com/HyukjinKwon/spark/actions).

cc @dongjoon-hyun @Yikun @gengliangwang @MaxGekk FYI

Seems working fine as expected.
We can merge this once https://github.com/HyukjinKwon/spark/actions/runs/2527942990 gets passed.
After this PR, developers should fetch and rebase their mater branch. I will send an PSA email to the dev mailing lists.
Other old branches won't be affected if I am not mistaken.

Seems working fine as expected.
We can merge this once https://github.com/HyukjinKwon/spark/actions/runs/2527981545 gets passed.
After this PR, developers should fetch and rebase their mater branch. I will send an PSA email to the dev mailing lists.
Other old branches won't be affected if I am not mistaken.

Guys I'll merge this soon and monitor today (it's morning my time) so I can follow up and fix some more if there are something going wrong

Merged to master.

+1, LGTM!

great job!

Late LGTM, thanks for the work!


After adding ingredient, focus does not go to textbox in next line

observed in mozilla broswer on linux machine

fixed: focus set on the name input box after adding ingredient, diet info

This issue was moved to k-sheth/calendar-khana-lagao#52


ppw init_dev failed

version 1.3.2
exception:
  EnvCommandError

  Command ['/home/aaron/miniconda3/envs/cheese/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--isolated', '--no-input', '--prefix', '/home/aaron/miniconda3/envs/cheese', '--no-deps', '/home/aaron/.cache/pypoetry/artifacts/a5/c6/8e/787aeae2abb2e89ea98a95e4a3122303096e22cdb2a47ef6a483e106d1/fire-0.4.0.tar.gz'] errored with the following return code 1, and output:
  Processing /home/aaron/.cache/pypoetry/artifacts/a5/c6/8e/787aeae2abb2e89ea98a95e4a3122303096e22cdb2a47ef6a483e106d1/fire-0.4.0.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'error'
    error: subprocess-exited-with-error

    × pip subprocess to install build dependencies did not run successfully.
    │ exit code: 1
    ╰─> [2 lines of output]
        ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)
        ERROR: No matching distribution found for setuptools>=40.8.0
        [end of output]

    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.

  at ~/miniconda3/envs/cheese/lib/python3.8/site-packages/poetry/utils/env.py:1540 in _run
      1536│                 output = subprocess.check_output(
      1537│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1538│                 )
      1539│         except CalledProcessError as e:
    → 1540│             raise EnvCommandError(e, input=input_)
      1541│
      1542│         return decode(output)
      1543│
      1544│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:

  PoetryException

  Failed to install /home/aaron/.cache/pypoetry/artifacts/a5/c6/8e/787aeae2abb2e89ea98a95e4a3122303096e22cdb2a47ef6a483e106d1/fire-0.4.0.tar.gz

  at ~/miniconda3/envs/cheese/lib/python3.8/site-packages/poetry/utils/pip.py:58 in pip_install
       54│
       55│     try:
       56│         return environment.run_pip(*args)
       57│     except EnvCommandError as e:
    →  58│         raise PoetryException(f"Failed to install {path.as_posix()}") from e

seems it's python-fire's issue

fixed in 0d6a86496984d27a3c5bc7b3544fb578f75a371a


Is it possible to specify a different context per entry point?

So I am working on a project that has 'N' number of entry points and we would like each entry point to be able to have a different version of React, for example. This would allow our development team to upgrade different pages to React v15.0.0 while leaving other pages running on older versions while still taking advantage of the ability to use the CommonsChunkPlugin.
Essentially we are looking for a way to have our cake and eat it too :)
Is this possible using webpack today? If it's not, would it be considered as a future feature?

Modules resolve relative to the requesting module. So you can create a directory structure like this:

app

node_modules (common stuff)
part1

node_modules (react A)
index1.js

part2

node_modules (react B)
index2.js

index1.js will use react A. index2.js will use react B.

Awesome thanks so much @sokra! We will try it out and let others know how this works out for us.

If this is still a problem, please open a question at Stack Overflow.


update dev-deps within semver, fixes  #79

Note this will likely need to be rebased off #80 (once it is merged) before merging, as tests were broken in master prior.

Rebased post-merge of #80.


Should we include manpages in VNM?

Discussed in https://github.com/NeuroDesk/neurodesk/discussions/83

Originally posted by civier August  4, 2021
The system has been minimised by removing packages and content that are not required on a system that users do not log into.
At least for me, it will be useful (not sure how much storage it occupies)

man-db

tried to do this, but the problem is that the man pages cannot be indiviudally reactivated and the unminimization would consume a lot of unesseary storage and increase the container size drastically.
Users should search online for the man pages


[BUG] WSAEFAULT

Problem
Brieftly to say, the erorr is overlapped  WSARecv call in gRPC that fails with a WSAEFAULT error.
What version of gRPC and what language are you using?
1.37.1
What operating system (Linux, Windows,...) and version?
windows 10
What runtime / compiler are you using (e.g. python version or version of gcc)
Python3.9 and VS2019 compiler
What did you do?
Run an sample ping-pong program with server side written by c++ and client side written written by csharp. let it run whole night.
What did you expect to see?
no errors
What did you see instead?
WSAEFAULT
I suspect if this line of code causes the bug
https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L290
according to the msdn api doc,

WSAEWOULDBLOCK | Windows NT:  Overlapped sockets: there are too many outstanding overlapped I/O requests. Nonoverlapped sockets: The socket is marked as nonblocking and the receive operation cannot be completed immediately.

is it handled correctly when the error is other than WSAEWOULDBLOCK ? so I am wondering if the right check here is  if (info->wsa_error == 0)
Also why not continuing reading more data when the received bytes == the recv buffer size in case there are more data in the socket buffer. there shoudl be a while loop for non-overlapped wsarecv until it got ewouldblock error.
I also suspect if this line of code causes the bug.
https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L299
according to msdn api doc,

lpNumberOfBytesRecvd
A pointer to the number, in bytes, of data received by this call if the receive operation completes immediately.
Use NULL for this parameter if the lpOverlapped parameter is not NULL to avoid potentially erroneous results. This parameter can be NULL only if the lpOverlapped parameter is not NULL.

So I am wondering if &bytes_read shoudl be changed to NULL.
Also I am wondering how this piece of code works ?
https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L192
why it is checking tcp->shutting_down instead of checking the retunred error code from wsarecv to detect the local/remote socket shutdown ?
Also why it does not apply the locker when read the value of tcp->shutting_down ? I can see it is protected with locker when set to true. https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L456
Also why not using the erro code from the next time wsarecv or wsasend to detect the local/remote socket abort/shutdown. The use of tcp-shutting_down looks a bit doggy.
Also there is inconsistency here:
https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L292
https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L311
finally I am wondering if it is correct to mix use of overlapped and non-overlapped wsarecv api ?
when you created the socket, I can see it is overlapped socket. but why the non-overlapped wsarecv is used here ?
https://github.com/grpc/grpc/blob/fd3bd70939fb4239639fbd26143ec416366e4157/src/core/lib/iomgr/socket_windows.cc#L188
Proposed Solution:
The way to operate with winsock in grpc looks a little bit doggy. the winsock apis are quite hard to make it right.
Please refer to the libuv as an example about how to operate winsock: either use overlapped or non-overlapped wsarecv.
overlapped  wsarecv: https://github.com/libuv/libuv/blob/9918a1743816dc49d6c664e41641d78ffd4a0705/src/win/tcp.c#L513
non-overlapped  wsarecv: https://github.com/libuv/libuv/blob/9918a1743816dc49d6c664e41641d78ffd4a0705/src/win/tcp.c#L1068
I will recompile the grpc with the proposed changes later and then let us see how it goes.

Hi @Kiddinglife. Following up, did you determine there was no bug here?

I am now testing it with 1.40.0 I can see there is bug fix I. That version regarding to windsock.

Not working.

Do you have the exact line of code from the logs or trace that triggered the WSAEFAULT?

@nicolasnoble please correct me if I'm wrong -
 I suspect if this line of code causes the bug
grpc/src/core/lib/iomgr/tcp_windows.cc

Line 290 in ea389c0

 if (info->wsa_error != WSAEWOULDBLOCK) {

according to the msdn api doc,
WSAEWOULDBLOCK | Windows NT:  Overlapped sockets: there are too many outstanding overlapped I/O requests. Nonoverlapped sockets: The socket is marked as nonblocking and the receive operation cannot be completed immediately.

is it handled correctly when the error is other than WSAEWOULDBLOCK ? so I am wondering if the right check here is if (info->wsa_error == 0)

We are only checking for WSAEWOULDBLOCK since that's the only case we would want to try an asynchronous read. For all other cases, the on_read callback should be able to deal with them.
I also suspect if this line of code causes the bug.
grpc/src/core/lib/iomgr/tcp_windows.cc

Line 299 in ea389c0

 &bytes_read, &flags, &info->overlapped, NULL);

according to msdn api doc,
lpNumberOfBytesRecvd
A pointer to the number, in bytes, of data received by this call if the receive operation completes immediately.
Use NULL for this parameter if the lpOverlapped parameter is not NULL to avoid potentially erroneous results. This parameter can be NULL only if the lpOverlapped parameter is not NULL.

So I am wondering if &bytes_read shoudl be changed to NULL.

I'm not sure about this but it doesn't seem to be used anyway, so it should be safe to use NULL. #27361
Also I am wondering how this piece of code works ?
grpc/src/core/lib/iomgr/tcp_windows.cc

Line 192 in ea389c0

 if (info->wsa_error != 0 && !tcp->shutting_down) {

why it is checking tcp->shutting_down instead of checking the retunred error code from wsarecv to detect the local/remote socket shutdown ?
Also why it does not apply the locker when read the value of tcp->shutting_down ? I can see it is protected with locker when set to true.

That does seem to be the case. @nicolasnoble do we need a lock here?
Also why not using the erro code from the next time wsarecv or wsasend to detect the local/remote socket abort/shutdown. The use of tcp-shutting_down looks a bit doggy.

The win_shutdown is an API for internal gRPC usage to let the socket library know that it is done with the socket and that all read/write callbacks should be failed.
Also there is inconsistency here:
grpc/src/core/lib/iomgr/tcp_windows.cc

Line 292 in ea389c0

 grpc_core::ExecCtx::Run(DEBUG_LOCATION, &tcp->on_read, GRPC_ERROR_NONE);

grpc/src/core/lib/iomgr/tcp_windows.cc

Line 311 in ea389c0

 grpc_socket_notify_on_read(tcp->socket, &tcp->on_read);

These achieve two separate things. In the first case, we already have a result from the read call and we just want to invoke the on_read callback to work on the result and in the second case grpc_socket_notify_on_read, the read is asynchronous and we are waiting on an IO notification that the read is done.
finally I am wondering if it is correct to mix use of overlapped and non-overlapped wsarecv api ?
when you created the socket, I can see it is overlapped socket. but why the non-overlapped wsarecv is used here ?

I believe the reason is a shorthand for performance reasons, so as to avoid an asynchronous path .

(Closed by mistake by clicking on Close with comment instead of Comment)

What I'm really asking is for a log extract so I can see more specifically what's going on here. I have no idea what line of code in our codebase can emit the "overlapped WSARecv call in gRPC that fails with a WSAEFAULT error" message for instance. So before actually committing to investigate further, I'd like to make sure we have enough context and data, which we don't.

Hey @nico  I am now also very confused about the error. you got efault error and then stream removed and disconnected. Nothing special I would say so. As I said, I am doubting if there is a buffer slice overwrite/corruption due to the potential race condition in iocp and background threads. So I am now doing the experiment with only using the first wsarecv to see if it works. Also it will be helpful if you can show me / check on your own the unit tests for buffer slices. The buffer slices look very like the iovec structure in Linux world. I am doubting if it is fully unit tested with memory corruption/overwrite cases. Thx.

also this piece of code:
  if (tcp->read_slices->length < DEFAULT_TARGET_READ_SIZE / 2 &&
      tcp->read_slices->count < MAX_WSABUF_COUNT) {
    // TODO(jtattermusch): slice should be allocated using resource quota
    grpc_slice_buffer_add(tcp->read_slices,
                          GRPC_SLICE_MALLOC(DEFAULT_TARGET_READ_SIZE));
  }

  GPR_ASSERT(tcp->read_slices->count <= MAX_WSABUF_COUNT);
  for (i = 0; i < tcp->read_slices->count; i++) {
    buffers[i].len = (ULONG)GRPC_SLICE_LENGTH(
        tcp->read_slices->slices[i]);  // we know slice size fits in 32bit.
    buffers[i].buf = (char*)GRPC_SLICE_START_PTR(tcp->read_slices->slices[i]);
  }

  TCP_REF(tcp, "read");

  /* First let's try a synchronous, non-blocking read. */
  status = WSARecv(tcp->socket->socket, buffers, (DWORD)tcp->read_slices->count,
                   &bytes_read, &flags, NULL, NULL);
  info->wsa_error = status == 0 ? 0 : WSAGetLastError();

I also doubt this piece of codes,
when tcp->read_slices->count == MAX_WSABUF_COUNT and tcp->read_slices->length < DEFAULT_TARGET_READ_SIZE / 2, does this piece of code have potentials to cause memory overwrite if some other pieces of code assume the tcp->read_slices->length must >= DEFAULT_TARGET_READ_SIZE  ?

Hi @Kiddinglife, @nicolasnoble said, if you can provide logs/traces that show the error that would be very helpful.

Hi @yashykt,
The error we see in the logs is the following:
[2021-9-20 14:4:43.597] GRPCPlugin(Runtime) [3016] INFO: OP[message_decompress:185D1400]:  CANCEL:{"created":"@1632164683.597000000","description":"Error in HTTP transport completing operation","file":"E:\GDK-VLT\Runtime\core\GDK\Externals\gRPC_1_40_0\src\core\ext\transport\chttp2\transport\chttp2_transport.cc","file_line":1236,"referenced_errors":[{"created":"@1632164683.597000000","description":"Attempt to send initial metadata after stream was closed","file":"E:\GDK-VLT\Runtime\core\GDK\Externals\gRPC_1_40_0\src\core\ext\transport\chttp2\transport\chttp2_transport.cc","file_line":1472,"referenced_errors":[{"created":"@1632164683.595000000","description":"Endpoint read failed","file":"E:\GDK-VLT\Runtime\core\GDK\Externals\gRPC_1_40_0\src\core\ext\transport\chttp2\transport\chttp2_transport.cc","file_line":2520,"grpc_status":14,"occurred_during_write":0,"referenced_errors":[{"created":"@1632164683.594000000","description":"OS Error","file":"E:\GDK-VLT\Runtime\core\GDK\Externals\gRPC_1_40_0\src\core\lib\iomgr\tcp_windows.cc","file_line":307,"os_error":"The system detected an invalid pointer address in attempting to use a pointer argument in a call.\r\n","syscall":"WSARecv","wsa_error":10014}]}]},{"created":"@1632164683.597000000","description":"Attempt to send trailing metadata after stream was closed","file":"E:\GDK-VLT\Runtime\core\GDK\Externals\gRPC_1_40_0\src\core\ext\transport\chttp2\transport\chttp2_transport.cc","file_line":1548}],"target_address":"ipv6:[::1]:50506"}

https://github.com/grpc/grpc/pull/27457 might be related

@yashykt - That looked promising but we were still able to reproduce the WSAEFAULT error on the overlapped WSARecv call even after removing the memset call (for #27457).

@nicolasnoble is driving this

I am concerned the fact that the WSABUF buffers[MAX_WSABUF_COUNT] is allocated on the stack is problematic. Could that explain the error reported by @cfandrich ?
https://github.com/grpc/grpc/blob/ea389c00c2c83e0364c5b1cfb73820c0101daa14/src/core/lib/iomgr/tcp_windows.cc#L248
I have similar concers for the WSASend side.

This issue has become a collection of Windows I/O bugs, some of which have been fixed. I'm going to focus exclusively on the WSAEFAULT and its reproduction cases. For any other issues, please file another iffue if they are still relevant.
@cfandrich can you create a minimal, complete example that reproduces this issue? If it is still reproducible on the latest gRPC version, I'll be curious to dig in. If you suspect use-after-free, and can't produce an example, then consider using some Windows tools (some built into visual studio) that can help catch various memory corruption problems. You could consider running your application with a JIT debugger enabled, or running it in a debugger with various detection tools enabled.

I am concerned the fact that the WSABUF buffers[MAX_WSABUF_COUNT] is allocated on the stack is problematic.

@Schramp I believe we discussed this on another issue, and it is fine per the Windows API spec.

I am concerned the fact that the WSABUF buffers[MAX_WSABUF_COUNT] is allocated on the stack is problematic.

@Schramp I believe we discussed this on another issue, and it is fine per the Windows API spec.

yes, we already discussed this (https://github.com/grpc/grpc/pull/28432#discussion_r1027319070), and you rightfully corrected me on my misunderstanding of the API. So you should ignore my remark on the stack allocation of buffers[MAX_WSABUF_COUNT]


PAT details

The README instructions state "You'll need to check the box for accessing repository data via the github API v4." -- I suggest this should be clearer as the settings for a new PAT (mine at least) don't mention v4 at all... I read this as instructing to check the box 'repository data via the github API v4' rather than just repo (or a subset).
Is the token environment variable name GITHUB_GRAPHQL_TOKEN a standard for something? devtools looks for GITHUB_PAT which I -- and presumably many others -- already have set. Is this specifically a new token or will that one (especially with that name) do the job? In either case the value gets read into session, after which point the name isn't needed, but it would be nice to only have one of those environment variables. My existing GITHUB_PAT, saved as GITHUB_GRAPHQL_TOKEN, seems to work.
Does stellar require repo, or user authorization? I created a new token with just user and saved it as GITHUB_GRAPHQL_TOKEN and this appears to work. My original GITHUB_PAT is already authorized for user so it also works.

created a team with write access to this repo

yep I think I remember Jeroen telling me GITHUB_PAT would work just as well, so I should probably change that next time I spend time on ghrecipes. I myself copied it directly from https://github.com/ropensci/ghql#initialize-client 👼

Upstream issue created: https://github.com/ropensci/ghql/issues/11

Great point on the ghql issue - ROpenSci is the one institution that can really get some momentum here in terms of nomenclative consistency.

I entirely overlooked that we're the ones enforcing GITHUB_GRAPHQL_TOKEN, not ghql. There's complete freedom to choose at the moment.
Proposal:
create_client could take an env_var parameter, passed from stars, default "", allowing a user to use an arbitrary environment variable as their token name. Otherwises searches GITHUB_GRAPHQL_TOKEN then GITHUB_PAT, taking the first it finds.
## use env_var with precedence, if present
## otherwise use GITHUB_GRAPHQL_TOKEN with precedence, if present
## otherwise use GITHUB_PAT, if present
## otherwise fail
get_token <- function(env_var = "") {
  token <- sapply(c(env_var, "GITHUB_GRAPHQL_TOKEN", "GITHUB_PAT"), Sys.getenv)
  token <- token[token != ""][1]
  if (is.na(token)) stop("NO TOKEN FOUND")
  token
}


cirq.MergeInteractions should also allow FSim and other future decompositions

Is your feature request related to a use case or problem? Please describe.
Currently the circuit optimizer cirq.MergeInteractions produces circuits with CZ (or parameterized CZ) gates by using cirq.two_qubit_matrix_to_operations() to decompose arbitrary two-qubit unitaries.  Now that decompositions exist for the FSim gate (cirq.decompose_two_qubit_interaction_into_four_fsim_gates() with a different API) and soon the sqrt-iSWAP gate, there should be a single or consistent API to access these.
I noticed this while working on #4083.
Describe the solution you'd like
MergeInteractions and two_qubit_matrix_to_operations could have a gate set parameter that lets them choose which decomposition to use.  It could simply be an argument gate=CZ or gate=FSim(theta, phi) but I'm not sure the best design.
Describe alternatives/workarounds you've considered
Write separate circuit optimizers for FSim and other primitive gates.
What is the urgency from your perspective for this issue? Is it blocking important work?

P3 - I'm not really blocked by it, it is an idea I'd like to discuss / suggestion based on principle

This is something @tanujkhattar is looking into as part of #3243 - in the long term, we'd like to migrate all of these into the decomposition API. Adding new decompositions before then is fine, but they'll also need to be migrated.


[WIP] feat: Separate Prover and Server DAL

What ❔

New db_connection crate, which contains all the main logic for managing ConnectionPool and StorageProcessor
Prover parts of DAL are moved to prover_dal crate
Now we have typed ConnectionPools and StorageProcessors, to use appropriate ConnectionPool there are 2 options:
ConnectionPool<Server>, ConnectionPool<Prover> - which will return appropriate StorageProcessor with corresponding methods accessible only by this connection pool.

Why ❔
To separate logic of prover and server. To prevent accessing wrong database with one ConnectionPool.
Checklist

[ ] PR title corresponds to the body of PR (we generate changelog entries from PRs).
[ ] Tests for the changes have been added / updated.
[ ] Documentation comments have been added / updated.
[ ] Code has been formatted via zk fmt and zk lint.
[ ] Spellcheck has been run via zk spellcheck.
[ ] Linkcheck has been run via zk linkcheck.

Although the PR is pretty big, here are some main points for reviewing(because PR is big mostly because of renaming/import changes):

db_connection crate - most of the logic moved there and generalized, although some of the metrics are exposed to be properly used in DAL crates. Also, Postgres metrics were left in the core dal crate, because they are accessing database.
Changes in zksync_types - prover types are moved from fri_prover_dal there, because they are being accessed from different crates
lib.rs files in core dal and prover_dal crates - that's where main logic for typed storage processors are implemented.

Also a few points worth discussion:

Right now dal crates are reexporting some types/traits from db_connection crate - it was done mostly because most of the crates are not using both prover and core DALs  - except witness generator and house keeper - not sure, but probably it is still better not to reexport types to prevent confusion.
global_config method in ConnectionPool seems to be static and not depending on type of ConnectionPool - it generally makes sense to move it out of the ConnectionPool struct, but not sure that it worth doing in this exact PR - probably we can do it in the other one.
Server/ServerProcessor and Prover/ProverProcessor struct can be generally united - but I don't think we want to use ConnectionPool<ServerProcessor> or ConnectionPool -> Server syntax
Any renaming suggestions are also appreciated

The problem with merges is that I need to fix it every few hours, because PR itself touches a lot of places and almost every merge to main causes conflicts. I will merge it for now, but I think will do it once a day if it will take much time for this one to merge


Fixed empty name error and also changed tests

I changed the error which the api returned when the metadata.name or generateName field is not provided. So now if you send in an OR separator between two fields, I changed prefix to take care of that case so that it shows that one of two field is required
#9284

@thockin

@quinton-hoole the associated issue was marked v1.0-post milestone. Can you confirm v1.0-post status for the PR?

Yes, I think that this v1.0-post.  @thockin can confirm.

This is a usability improvement. We could take it for 1.0.

That said, it requires a bit more thought before we accept it. Will comment more on the issue.

Yeah @manansaraf and I struggled a lot with the design. Happy to change it in any way.

Trying to clear the PR queue...
I considered oneOf/OneOf, with some kind of set syntax, and |. Since this will be read by humans, I agree that English is better.
So, one minor nit: Make it or rather than OR, similar to label selector operators.
cc @lavalamp in case he can think of any problems with this, and to find someone to finish the review
Please rebase.

This looks fine to me.

LGTM, thanks!
Needs rebase before this can be merged.

@manansaraf needs rebase

Sorry just saw this will rebase and push a new commit soon

Rebased and changed "OR" to "or" :)

@k8s-bot ok to test

LGTM

@k8s-bot test this [testing build queue, sorry for the noise]

Automatic merge from SubmitQueue


Cantrip class

Instead of a all characters having the same free active, replace it with a new Cantrip interface, from which the basic attack inherits.
A Cantrip is different from an AbstractActive in that it never costs mana.

Removed, as it can easily be replicated by setting an AbstractActive's mana cost to 0.


Avoid sending empty root hints list

This PR need to wait for https://github.com/fortanix/rustls/pull/5 to be merged.
Then update the rustls to use new tag again.

Sry, accidentally create this wrong PR!


Release a netstandard compatible version

We'd like to use this library in a netstandard1.3 project, but the latest version on Nuget only supports netcoreapp1.0. Would it be possible to release another beta version with the netstandard support in the master branch?

@jonjomckay, sure a new version will be released soon, to support netstandard1.3 and netstandard2.0

The latest release now supports netstandard1.3, so I will close for now, please reopen if you have any other concerns

Awesome, thanks @ahmad-moussawi!