Add MSPatchInfo to the file spec

The contents go like this:
MSPatchInfo {
  lastIntegratedPatchID: string
  statusInfo: MSCollaborationStatusInfo {
    status: string
    uploadedPatches: array of strings
    isConsolidating: bool
    shareID: string
    baseVersionID: string
    versionID: string
  }
  MSPatchContainer {
    localPatches: an array of MSPatch objects
  }

This object contains the information we need to track the state of collaboration documents and should really only be present in documents within the hidden cloud folder on users machines.
The contents of MSPatch objects are described in Notion
@jedrichards hope that's useful. I'm not sure how all this stuff works, so ping me if you need more info.

Thanks @opsGavin! What would be even better is if you could also dump some JSON here, as it appears on disk? Since that's the data that is being described by the schemas in this repo. I note that _class strings like MSImmutablePatchInfo appear on disk, but are not mentioned in your description above.

Sorry @jedrichards I'm not ignoring this. We're looking at making some changes here, but if you bear with me I'll come back and add some examples when that's done.

No problemo, sounds good 👍

Wondering if this is no longer relevant now we've added documentState?

We're still going to need this for co-edit in some form, but I'm not certain how it will look in the end.

@opsGavin is this still relevant?

Yes - still relevant and probably a little more stable now. Here's an example of the patchInfo from a document I have here.
  "patchInfo" : {
    "lastIntegratedPatchID" : "f6722512-1c0c-4a21-b016-31e84762652c",
    "status" : {
      "statusString" : "COLLABORATING",
      "_class" : "SharedEditing.MSCoEditStatusInfo",
      "shareID" : "30aa41e9-f735-4696-b005-263a8df6da47"
    },
    "_class" : "MSImmutablePatchInfo",
    "localPatches" : [
      {
        "_ref_class" : "MSPatch",
        "_class" : "MSJSONFileReference",
        "_ref" : "patches\/01d7c53c-d446-4371-ae84-6b023327d659"
      },
      {
        "_ref_class" : "MSPatch",
        "_class" : "MSJSONFileReference",
        "_ref" : "patches\/9f759e56-349f-40ae-8ae4-b3b4fd1cbdcc"
      },
      {
        "_ref_class" : "MSPatch",
        "_class" : "MSJSONFileReference",
        "_ref" : "patches\/a6e9bb74-00e3-46d8-acfc-d2f295b115ca"
      }
    ]
  },

The status contains information about the document which is basically just the shareID along with a string describing whether a connection was established. I wouldn't expect to see anything other than COLLABORATING here although I think LOCAL or INITIAL may be possible in theory (/cc @mkeiser).
localPatches contains references to patch files which have still to be uploaded to the server. Normally there won't be any, but if the user closes the document while off-line there could be and these will be sent when they re-open the document.
Details of the contents of the patch files can be found at https://www.notion.so/sketchapp/Overview-6f3f6dd858e74dd084c08e6b5e24b4b6#d95b4faaaa434714826d6f10d97efd6e. I believe there is also a schema for them somewhere - @mz2 would probably be able to point you to that.
It's worth just re-iterating that this information should only be found in cloud documents which should be hidden from the user and that we certainly don't want anyone trying to edit any of this information.
Shout if you have any questions

Still feels like we haven't come to a decision on this one. Since patchInfo is by all accounts something of an implementation detail, does it even belong in the public schemas?

One of the mentioned outcomes was to document it here, not because it's useful for third parties (it isn't) but to remain documented.

I'd like to unblock this if possible. If we want to go ahead with it there are still one or two todos on the PR, e.g. https://github.com/sketch-hq/sketch-file-format/pull/115#pullrequestreview-488556820.
That said, I'd still lean towards perhaps closing it instead. I understand the desire to document it, but since it isn't really a public-facing property, I assume it's documented already where it needs to be - in the Sketch code base? Does it make sense to document it here too, on the public schema repo, where it could easily get out of date etc?

@christianklotz This has arrived in base now, so someone could take a look at it, but still seems to the way forwards isn't that clear.
I'm still leaning towards closing it to be honest, but are you are keen to press ahead and document the collab patch format in the public schemas?
Although closed, the schema changes in the linked PR are probably still relevant, although would be worthwhile to get some Mac dev 👀 on them to see if they are still correct.

I think this is the version that we shipped in 71 (cc @opsGavin to double check):
    "patchInfo": {
        "_class": "MSImmutablePatchInfo",
        "baseVersionID": "72704274-de94-4c29-b04a-984694d652f0",
        "lastIntegratedPatchID": "1a8f3c2a-8b32-4f78-8d75-3ecbf2d2c07a",
        "localPatches": [
            {
                "_class": "MSJSONFileReference",
                "_ref": "patches/151bdc13-f065-4772-8299-32e92caf05c5",
                "_ref_class": "MSPatch"
            }
        ],
        "receivedPatches": [
            {
                "_class": "MSJSONFileReference",
                "_ref": "patches/6d1dd1b4-9323-42d2-9670-e2b57a8e52df",
                "_ref_class": "MSPatch"
            }
        ]
    }


Cannot Construct a Matrix4X4 with swapped near and far plane

So this isn't necessarily a bug, but it prevents a particular feature from being implemented. Namely Reversed-Z Depth Buffering. In order to implement a reversed depth buffer, one must swap the values of the near and far plane when constructing a projection matrix. When attempting to do so with Matrix4X4.CreatePerspectiveFieldOfView an ArgumentOutOfRangeException is raised. Now, when using a standard depth buffer and/or when doing other operations this is a perfectly legitimate exception.  So, on the one hand I understand why it's there, but some alternative is needed in order to not block the above use case.
Just removing that specific exception clause works and seems to be the simplest option. In that case we simply hope that if a user mistakenly uses a smaller far plane than near plane, rather than intentionally flipping it, that the issue will be self-apparent. The other option is to make some type of method that takes a projection matrix and "reversifies" it. Not sure which is more preferable or if there's another option I'm missing.

Thank you for the report. I'll look into this.

@HurricanKai Okay, so after getting some sleep I think I've found a simple way to do this that keeps everyone happy. It implements the feature in an opt-in manner, maintains the existing exception checks and should not break existing code that relies on the API. PR has been opened.

Closed by #504


Bugfix for BETWEEN method and regex fix for BETWEEN and WHERE methods.

Fix the problem with where and between when use a function sql

@pjeldres Corrije esos puntos y estamos ok


Please export Blob class and BUFFER symbol

Hi,
I'm using node-fetch with https://github.com/RangerMauve/webrun/ and running into problem because Blob class can't be referenced nor there is a way to extract bytes from it. Please consider exposing so it could be used.

Closing as duplicate of #392


another error about extracting the skeleton of diving48

Hi！Kenny！ Thank you for sharing such excellent work！I have met another error when extracting the skeleton of diving48. I felt very confused.
I can run demo successfully.
However, when I run custom_2d_skeleton.py, there is an error about init_detector and init_pose_model
If I use the original code

If I modify the default_det_config and default_det_ckpt according demo_skeleton.py

I really don't know how to solve the problem, if anyone meet the same problem like me , please help me, thank you.

Hi, please install the mmdet & mmpose from source, with the development mode, like pip install -e .

@kennymckormick Thanks a lot, I will have a try.


add text only mode

if want to show a text without image,now can use:
+ (void)showTextWithStatus:(nullable NSString*)status;

I take over #811 and #813 ,I create it
Can you merge this PR? @honkmaster

I can't see the corresponding API + (void)showTextWithStatus:(nullable NSString*)status; in swift version while there is no way to realize a text only function on HUD.

pod 'SVProgressHUD',:git => 'https://github.com/Fidetro/SVProgressHUD.git'

@DeforeChen you can use SVProgressHUD.showText(withStatus: "xxx") in swift

pod 'SVProgressHUD',:git => 'https://github.com/Fidetro/SVProgressHUD.git'

@DeforeChen you can use SVProgressHUD.showText(withStatus: "xxx") in swift

SVProgressHUD.showText(withStatus: "what")
While that cause a crash as follows:
+[SVProgressHUD showTextWithStatus:]: unrecognized selector sent to class
By the way, in my project, I use Swift 4

@honkmaster any plans to add this?

The PR is incomplete, an update for the demo would be required.

@honkmaster I have updated demo,please review pr

@honkmaster I updated readme and demo,any other questions?

@honkmaster wating for...

@honkmaster wating for...

It's like that in swift , we can use + (void)showImage:(UIImage*)image status:(NSString*)string; because the image is nonull
And there is no way to set the imageview hidden or nil
If I set the imagesize to zero , the text is not in the center

@honkmaster


grpc_ares_ev_driver_windows: Overlapped I/O operation is in progress

What version of gRPC and what language are you using?
Updated today to dotnet 2.32 in hope to fix this issue, however had the same experience on 2.27.
What operating system (Linux, Windows,...) and version?
Windows Server 2016  (Azure App Service)
What runtime / compiler are you using (e.g. python version or version of gcc)
.NET 4.7.1
What did you do?
I'm only using gRPC client requests in my ASP.NET project, and from there I'm fetching info from an external gRPC API. The current implementation has been working flawlessly for a couple of months, however since some time back all of these gRPC requests are insanely slow.
What did you expect to see?
Normal response times < 1 second. The external API works just as before which can be verified by that the same requests are still fast when running the same code locally on my machine.
What did you see instead?
Response times are now around 90 seconds for the very same requests that works normally when running locally on my windows machine. Something on the Azure App Service environment are slowing these requests down greatly, it seems to be the DNS resolution.
From what I can tell from the logs, it pretty quickly shows this:
T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:185: (c-ares resolver) fd:|c-ares socket: 4588| RegisterForOnReadableLocked WSARecvFrom error code:|997| msg:|Overlapped I/O operation is in progress

Then it takes about 90 seconds before it calls it a day with:
T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc:604: (c-ares resolver) fd:|c-ares socket: 4588| OnIocpReadableInner winsocket_->read_info.wsa_error code:|995| msg:|{""created"":""@1600945877.332000000"",""description"":""OS Error"",""file"":""T:\src\github\grpc\workspace_csharp_ext_windows_x86\src\core\ext\filters\client_channel\resolver\dns\c_ares\grpc_ares_ev_driver_windows.cc"",""file_line"":599,""os_error"":""The I/O operation has been aborted because of either a thread exit or an application request.\r\n"",""syscall"":""OnIocpReadableInner"",""wsa_error"":995}|",

After that, handshake is performed, request is made and a response is received correctly.
Anything else we should know about your project / environment?
I also have another .NET Core project in the same hosting environment on Azure, and everything is still running correctly there with grpc-dotnet.

I tried using GRPC_DNS_RESOLVER=native however the results are pretty much the same so I'm guessing it has nothing to do with ares itself.  But I cannot figure out as to why it cannot resolve dns in my azure environment

@apolcyn  can you take a look?

A packet trace on the DNS server port may help to give some insight here.
For example, if you could run the client and sniff all UDP and TCP traffic with source and/or destination port 53, then we could see the which DNS queries are being sent and when, and which DNS responses are being received and when. That may give insight to e.g. possible unavailability of the nearby DNS server.
Also, it looks like you're already ran the code under verbose grpc trace logs. Can you please share those logs here?
The minimal grpc logs to get could be captured with these env vars:
GRPC_VERBOSITY=DEBUG
GRPC_TRACE=cares_resolver,cares_address_sorting (this will give verbose logs assuming you're using the c-ares resolver).
Another way to go about debugging this could be to remove gRPC from the equation and test DNS resolution in isolation. For example, something along the lines of the following python command could be ran from a shell:
python -c "import socket; socket.getaddrinfo('my.problematic.server', 80, 0, 0, socket.IPPROTO_TCP)"

That would test the system resolver (the same resolver as what's used when gRPC is ran with GRPC_DNS_RESOLVER=native)
If that reproduces the DNS resolution issue, a packet sniffer ran in the background could give some insight.

Thank you for looking into this!
I was actually not able to overcome this issue at all and due to features needing to be published asap I chose to migrate the API to .NET Core where I knew it would work. It now works fine with grpc-dotnet in my case since there is no dependency to grpc_ares_ev_driver_windows.
Because of this I'm also no longer able to reproduce this error and it can be closed if you wish so.


feat: update SDK to v0.43.0 and add upgrade handler

Update cosmos SDK version to v0.43.0-rc2
Add upgrade handler with name v0.43.0-upgrade

TODO: Need to test upgrade thoroughly and update upgrade handler if needed any changes.

Closing this PR as these changes are already implemented in https://github.com/ovrclk/akash/pull/1343


[WIP] Revise nx.compat to improve the capability of reuse NetworkX code

What do these changes do?

[x] Add a update method to NodeDict to fix forward networkx.convert_node_labels_to_integers problem;
[ ] Pass forward algorithm ci;
[ ] Enhance nx.compat layer, decrease the import time of graphscope.nx

Related issue number

Fixes #1502

Codecov Report

Merging #1584 (d7dafbf) into main (55d31d8) will decrease coverage by 32.89%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #1584       +/-   ##
===========================================
- Coverage   73.17%   40.28%   -32.90%
===========================================
  Files          88       88
  Lines        9714     9714
===========================================
- Hits         7108     3913     -3195
- Misses       2606     5801     +3195

Impacted Files
Coverage Δ

python/graphscope/tests/unittest/test_java_app.py
0.00% <0.00%> (-100.00%)
:arrow_down:

...ython/graphscope/tests/unittest/test_cython_ast.py
0.00% <0.00%> (-100.00%)
:arrow_down:

...ython/graphscope/tests/unittest/test_serailaize.py
0.00% <0.00%> (-100.00%)
:arrow_down:

python/graphscope/analytical/udf/patch.py
3.47% <0.00%> (-96.53%)
:arrow_down:

python/graphscope/tests/unittest/test_lazy.py
0.00% <0.00%> (-96.30%)
:arrow_down:

...thon/graphscope/tests/unittest/test_scalability.py
0.00% <0.00%> (-96.16%)
:arrow_down:

python/graphscope/tests/unittest/test_app.py
0.00% <0.00%> (-95.66%)
:arrow_down:

...hon/graphscope/tests/unittest/test_create_graph.py
0.00% <0.00%> (-92.00%)
:arrow_down:

python/graphscope/tests/unittest/test_graph.py
0.00% <0.00%> (-84.93%)
:arrow_down:

python/graphscope/tests/unittest/test_context.py
0.00% <0.00%> (-81.04%)
:arrow_down:

... and 38 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 55d31d8...d7dafbf. Read the comment docs.


StatusBar Fonts color

how to set  statusBar fonts color?

This is a question for Ionic proper: https://github.com/driftyco/ionic


react-scripts test is watching files in node_modules

Can you reproduce the problem with latest npm?
yes
Can you still reproduce it?
yes, also did npm cache clean just to be sure
Description
This is a followup of "npm test hangs for newly generated app (#960)". After reading the comment of @hmeerlo that "Jest is trying to watch too many files at once (https://github.com/facebook/jest/issues/1767#issuecomment-250719565)" I suspect that files in node_modules are unnecessarily being watched.
Expected behavior
npm t should run without error.
Actual behavior
react-scripts test --env=jsdom

2017-05-28 09:18 node[1154] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22)
2017-05-28 09:18 node[1154] (FSEvents.framework) FSEventStreamStart: register_with_server: ERROR: f2d_register_rpc() => (null) (-22)
events.js:160
      throw er; // Unhandled 'error' event
      ^

Error: Error watching file for changes: EMFILE
    at exports._errnoException (util.js:1018:11)
    at FSEvent.FSWatcher._handle.onchange (fs.js:1420:11)
npm ERR! Test failed.  See above for more details.

Environment
Run these commands in the project folder and fill in their results:

npm ls react-scripts (if you haven’t ejected<EMAIL_ADDRESS>node -v: v6.10.3
npm -v: 4.6.1

Then, specify:

Operating system: macOS Sierra 10.12.5
Browser and version: -
watchman never installed

Reproducible Demo
https://github.com/thisconnect/cra-desktop
Reproduce the error
This is an instantiated create-react-app (un-ejected) with electron added as described in PR https://github.com/facebookincubator/create-react-app/pull/1718
git clone https://github.com/thisconnect/cra-desktop
cd cra-desktop
npm install
npm test

Without error (when removing 2 large dependencies)
Assuming react-scripts test is indeed watching files in node_modules, remove electron and electron-packager from devDependencies in package.json, then rm -rf node_modules && npm i && npm t, test mode runs fine. This does not yet prove that files in node_modules are being watched.
Strong indication that node_modules is being watched:

run any CRA instance without error
the Terminal should display something like "No tests found related to files changed since last commit" and the "Watch Usage"
open a random .js file in node_modules
for example node_modules/mime/mime.js
watch the Terminal (screenshot 1)
add a comment on any line i.e. // test
save the file
check if the terminal changed (screenshot 2)

Terminal before changing a random file in node_modules (screenshot 1)

Terminal after chaning a random file in node_modules (screenshot 2)

Side note: The error does not appear in<EMAIL_ADDRESS>and was introduced with<EMAIL_ADDRESS>

I am not exactly sure what the corresponding jest config is, maybe testMatch in https://github.com/facebookincubator/create-react-app/blob/master/packages/react-scripts/scripts/utils/createJestConfig.js ?
My test about react-scripts test watching seems to not only affect node_modules/**/*.js, but also files in /dist/*.js and in /build/*.js.

Thanks for creating the example source @thisconnect - I have a PR in that should fix this issue.
Although the underlining issue is with Jest, and will depend if the maintainers of CRA would prefer to wait for Jest to fix, or fix it here until it's fixed in Jest.

How hard is it to fix in Jest?

I am really unsure.
As mentioned in facebook/jest/issues/1767 when Jest is using the node_watcher it is tells it to watch everything with chosen extensions, rather than just the files in testMatch.
Its possible the fix is just to pass testMatch to the file watcher.  Unfortunately I don't understand why Jest does what it does. It might just be a bug or if there is some greater reason for it.
Hopefully @cpojer or @pugnascotia will have a better answer.

@ro-savage fyi, if Jest only watches the files matching testMatch, that means that we won't be able to run tests when non-test files change. So that is the expected behavior.

@cpojer - Of course, it didn't even click that it was only matching the test files.
Then it should be fine to use both testMatch and modulePathIgnorePatterns?
In fact modulePathIgnorePatterns would be the way correct to tell Jest what files/folders not to watch?
(I understand there might be edge cases where you npm install an updated module and then want the tests to rerun. But seems better to miss that edge case than crash.)

This might help fix the crashes: https://github.com/amasad/sane/pull/97.
I think we should first direct effort there, and if it doesn’t work out, try strategies in this issue.

New sane version works for me https://github.com/amasad/sane/pull/97#issuecomment-309203580

Jest merged sane in https://github.com/facebook/jest/commit/acc783a1cec93d484b72dd10e6b6192fc53194d4

Reinstalling watchman fixed this error for me.
brew uninstall watchman
brew install watchman

<EMAIL_ADDRESS>should have the fixed sane using fsevents
https://github.com/facebook/jest/issues/1767#issuecomment-313434888

I did a sudo port install watchman, and problem went away.
Apparently you need watchman installed.

@gaearon just FYI I am testing<EMAIL_ADDRESS>(without watchman) and it is actually the first time npm test runs with the electron stuff :tada:  :confetti_ball: :violin: :trophy: :guitar: :cocktail: :birthday: :rainbow: :rocket:  (test repo https://github.com/thisconnect/cra-desktop)
Had a minor issue running npm start, commented in https://github.com/facebook/create-react-app/issues/3815#issuecomment-385618503 and another issue with node6 travis CI https://travis-ci.org/thisconnect/cra-desktop/builds/373373379 something about Requires Babel "^7.0.0-0", but was loaded with "6.26.0".
(travis node8 is fine)
I am sorry for commenting on this closed issue, just wanted to report and say thanks. Thanks!


Popular Function Not Working

Site Link
(feed in the footer)
It seems to work great for traditional shots, but when I try to bring in popular it breaks and gives me a console error. Here is my code and a screenshot of my error. The error is still present on the live link.
jribbble.popular({token: "637f6fe822b65aa05a6fc2e91230202a2318716c5d5e7ec4c018b8bfd41af23f",page:1, per_page:48}, function(shotsArray) {   document.querySelector(".dribbble-shots-list").innerHTML = shotsArray.reduce(function(html, shot) {     return html + '

';   }, ""); });

I'm gonna go ahead and close this. We ended up chatting over email, but I'll repeat what we discussed there for anyone else that hits this issue in the future:

The reason popular shots is failing is because that's a special resource in the Dribbble API. It's only available to apps with Dribbble's approval. I forgot I had that in the docs here https://github.com/tylergaw/jribbble#methods-that-will-only-work-with-dribbble-approved-apps and Dribbble has it in their API docs here http://developer.dribbble.com/v2/shots/#list-popular-shots


Fix checksum error type

We have a couple of checksum error problems;

Unused checksum related variants in CharError
Unuseful error message (duplicated string "invalid checksum")

Fix both at the same time by removing the unused variants from CharError and by renaming the InvalidChecksum variant to InvalidResidue and improving the Display impl appropriately.

Hope you weren't still reviewing, I force pushed the string change as suggeseted.


Check status of our MDX components documented on site

We currently document our custom MDX components with examples. Check on their status, have we missed any, are they up to date? Do they work? 👯♀
See https://60e2f04459ef0300072d8771--prisma2-docs.netlify.app/docs/about/style-guide/mdx-examples

All good


How to pass other type argument to aws lambda function invoked by aws sdk c++

Confirm by changing [ ] to [x] below:

[x] I've gone though Developer Guide and API reference
[x] I've searched for previous similar issues and didn't find any solution

Platform/OS/Hardware/Device
What are you running the sdk on?
win 10
Describe the question
Local end I am working on aws sdk c++ and at cloud end, it is the python lambda function.
How can I pass other type arguments like OpenCV Mat data into the python lambda function to process it and return the processed data to local end?
I saw the example on https://docs.aws.amazon.com/code-samples/latest/catalog/cpp-lambda-lambda_example.cpp.html, it seems they are string type and in json style.
jsonPayload.WithString("key1", "value1");
  jsonPayload.WithString("key2", "value2");
    jsonPayload.WithString("key3", "value3");

Logs/output
If applicable, add logs or error output.
To enable logging, set the following system properties:
REMEMBER TO SANITIZE YOUR PERSONAL INFO
options.loggingOptions.logLevel = Aws::Utils::Logging::LogLevel::Trace;
Aws::InitAPI(options)

@GitZinger ,
Can you provide minimal sample code on what you're trying to do and which specific call you'd like to change, as is the description is kind of broad and I might suggest the wrong thing depending on the situation....
For example, I could say that you send the same type of json type and instead of having:
jsonPayload.WithString("key1", "value1");
you'd have to convert your CV::Mat to double and then send it:
jsonPayload.WithDouble("key1", 3.41);
But without seeing what you're trying to do I'm not 100% sure that's correct.

Yes. But I guess both the arguments and response or return have the limitation of the data. So eventually it is not working. it has to work with S3. Thank you


Mac os x vagrant install fails when using vagrant > 1.9.2

The vagrant cookbook computes the mac extension as .dmg.  For vagrant 1.9.2 the extension changed to _x86_64.dmg.  Needs a fix in the upstream vagrant cookbook.

https://github.com/jtimberman/vagrant-cookbook/pull/79 for the fix.


Corrupted values in precip fields

There are evidently some corrupted values in the gpcp precip fields:
import xarray as xr

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})
precip_ts = ds.precip.mean(("latitude", "longitude"))
precip_ts.plot()

It is unclear to me at this point whether these are in the original dataset (seems unlikely) or are instead an artifact that was generated in our pipeline.

Here is the mask I am using to filter out the corrupted data
mask = (ds.precip >= 0) & (ds.precip < 1000)

The corruption appears to occur independent of execution framework (i.e., it is not to_beam-specific). Using the current production dataset, we can zoom in on an area containing two corruptions:
import xarray as xr

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})
precip_ts = ds.isel(dict(time=slice(2150, 2250))).precip.mean(("latitude", "longitude"))
precip_ts.plot()

Then, in a notebook, we can clip the recipe's file_list to just this window using the same slice:
file_list = file_list[2150:2250]

Constructing a recipe from this clipped file_list and then running it in the notebook with recipe.to_function()() produces the same corruptions we see in the production data:
recipe.to_function()()
ds = xr.open_dataset(recipe.target_mapper, engine="zarr", chunks={})
precip_ts = ds.precip.mean(("latitude", "longitude"))
precip_ts.plot()

This actually does appear to originate in the source data itself. Zooming in, we can see that the leftmost corruption in the plots shown the previous comment occurs at index 2167:
import xarray as xr

store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})
precip_ts = ds.isel(dict(time=slice(2166, 2168))).precip.mean(("latitude", "longitude"))
precip_ts.plot()

We can then pull that specific source file out of the file_list
file_list[2167]
# --> 'https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/2002/gpcp_v01r03_daily_d20020907_c20170530.nc'

Download it
$ wget https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/2002/gpcp_v01r03_daily_d20020907_c20170530.nc

And open it to see that the mean precipitation for that file is indeed -28007.459
ds = xr.open_dataset("gpcp_v01r03_daily_d20020907_c20170530.nc")
ds.precip.mean(("latitude", "longitude"))

THIS IS WHY WE NEED PANGEO FORGE! 🚀

From Howard Diamond at NCEI via email:

GPCP data at NCEI are mirrored from what we receive from NASA via https://www.nccs.nasa.gov/services/data-collections/atmosphere-based-products/gpcp; so, if there are any corrupted data, that would come to us from NASA; if NASA corrects that, then we would post whatever corrections they may make.  We do no processing of the GPCP data.

Hello! I would like to track down where this data is coming from NASA, but I'm afraid that I don't recognize this dataset. How do I find out more about what this dataset is? I don't suppose you have a DOI for it? Or a product name and version?

@christine-e-smit - thanks a lot for responding here! We acquired the data from NOAA NCEI and have faithfully reproduced the license information and provenance details that were available on the NCEI website:
https://github.com/pangeo-forge/gpcp-feedstock/blob/78ebfa39faafbb3eb352b3cdc0856d4b1a722b95/feedstock/meta.yaml#L12-L17
As you can see, the NCEI website (https://www.ncei.noaa.gov/products/global-precipitation-climatology-project) does not provide a DOI.
This information is what appears on the corresponding Pangeo Forge page: https://pangeo-forge.org/dashboard/feedstock/42

Digging deeper in that page, under the "dataset" tab, I can see the following metadata embedded in the dataset
Conventions :
CF-1.6, ACDD 1.3
Metadata_Conventions :
CF-1.6, Unidata Dataset Discovery v1.0, NOAA CDR v1.0, GDS v2.0
acknowledgment :
This project was supported in part by a grant from the NOAA Climate Data Record (CDR) Program for satellites.
cdm_data_type :
Grid
cdr_program :
NOAA Climate Data Record Program for satellites, FY 2011.
cdr_variable :
precipitation
comment :
Processing computer: eagle2.umd.edu
contributor_name :
Robert Adler, Jian-Jian Wang
contributor_role :
principalInvestigator, processor and custodian
creator_email :
<EMAIL_ADDRESS>creator_name :
Dr. Jian-Jian Wang
date_created :
2017-05-30T16:52:42Z
geospatial_lat_max :
90.0
geospatial_lat_min :
-90.0
geospatial_lat_resolution :
1 degree
geospatial_lat_units :
degrees_north
geospatial_lon_max :
360.0
geospatial_lon_min :
0.0
geospatial_lon_resolution :
1 degree
geospatial_lon_units :
degrees_east
history :
1) 2017-05-30T16:52:42Z, Dr. Jian-Jian Wang, U of Maryland, Created beta (B1) file
id :
199610/gpcp_v01r03_daily_d19961001_c20170530.nc
institution :
ACADEMIC > UMD/ESSIC > Earth System Science Interdisciplinary Center, University of Maryland
keywords :
EARTH SCIENCE > ATMOSPHERE > PRECIPITATION > PRECIPITATION AMOUNT
keywords_vocabulary :
NASA Global Change Master Directory (GCMD) Earth Science Keywords, Version 7.0
license :
No constraints on data access or use.
metadata_link :
gov.noaa.ncdc:XXXXX
naming_authority :
gov.noaa.ncdc
platform :
GOES (Geostationary Operational Environmental Satellite), GMS (Japan Geostationary Meteorological Satellite), METEOSAT, TIROS > Television Infrared Observation Satellite, DMSP (Defense Meteorological Satellite Program)
processing_level :
NASA Level 3
product_version :
v01r03
project :
GPCP > Global Precipitation Climatology Project
publisher_email :
<EMAIL_ADDRESS>publisher_name :
NOAA National Centers for Environmental Information (NCEI)
publisher_url :
https://www.ncei.noaa.gov
references :
Huffman et al. 1997, http://dx.doi.org/10.1175/1520-0477(1997)078<0005:TGPCPG>2.0.CO;2; Adler et al. 2003, http://dx.doi.org/10.1175/1525-7541(2003)004<1147:TVGPCP>2.0.CO;2; Huffman et al. 2009, http://dx.doi.org/10.1029/2009GL040000; Adler et al. 2017, Global Precipitation Climatology Project (GPCP) Daily Analysis: Climate Algorithm Theoretical Basis Document (C-ATBD)
sensor :
Imager, TOVS > TIROS Operational Vertical Sounder, SSMI > Special Sensor Microwave/Imager
source :
/data1/GPCP_CDR/GPCP_Output/1DD//bin/199610/stfsg3.19961001.s
spatial_resolution :
1 degree
standard_name_vocabulary :
CF Standard Name Table (v41, 22 February 2017)
summary :
Global Precipitation Climatology Project (GPCP) Daily Version 1.3 gridded, merged satellite/gauge precipitation Climate data Record (CDR) from 1996 to present.
time_coverage_duration :
P1D
time_coverage_end :
1996-10-01T23:59:59Z
time_coverage_start :
1996-10-01T00:00:00Z
title :
Global Precipitation Climatatology Project (GPCP) Climate Data Record (CDR), Daily V1.3

If you want to play with the data yourself, you can do it with the following python code
import xarray as xr
store = 'https://ncsa.osn.xsede.org/Pangeo/pangeo-forge/gpcp-feedstock/gpcp.zarr'
ds = xr.open_dataset(store, engine='zarr', chunks={})

What is confusing to me here is that NCEI says "GPCP data at NCEI are mirrored from what we receive from NASA". However, nowhere in the NCEI metadata or website is there any mention of this provenance. The NCEI website suggests that the data come from University of Maryland.

Thanks @rabernat! Let me see what I can track down.

It looks as though NASA's GES DISC (where I work) archives several GPCP datasets: https://disc.gsfc.nasa.gov/datasets?keywords=gpcp&page=1. Unfortunately, none of these seem to match the temporal and spatial resolution of your dataset. So I've submitted a ticket to our help desk to see if I can find someone who is familiar with the GPCP project as a whole and might be able to help.

Who is responsible for this data?
Just as I'd hoped, one of our data support scientists, Zhong Liu, was able to solve the mystery of what this data's relationship is with NASA. It seems that you are using an old version of this data and the PI, Bob Adler, has retired from NASA. Since the GES DISC is not the archive for this data, we will contact Bob and the data curator.
Why are there negative values in precipitation?
Zhong, @briannapagan, and I took a quick look at a granule for one of the problematic dates, gpcp_v01r03_daily_d20020907_c20170530.nc, which we downloaded from https://www.ncei.noaa.gov/data/global-precipitation-climatology-project-gpcp-daily/access/2002/. Our best guess is that the fill value was set incorrectly. The metadata says the fill values is -9999, but it looks like this file is accidentally using -99999:
In [1]: import xarray as xr

In [2]: import numpy as np

In [3]: ds = xr.open_dataset("gpcp_v01r03_daily_d20020907_c20170530.nc")

In [4]: np.where(ds.precip<0)
Out[4]:
(array([0, 0, 0, ..., 0, 0, 0]),
 array([  2,   2,   2, ..., 177, 177, 177]),
 array([180, 181, 182, ...,  57,  58,  59]))

In [5]: ds.precip[0][2][180].values
Out[5]: array(-99999., dtype=float32)

In [6]: ds.precip[0][177][58].values
Out[6]: array(-99999., dtype=float32)

How can I get the newest version of this dataset?
The GES DISC archives an updated version of this collection. We have both a daily and a monthly product: https://disc.gsfc.nasa.gov/datasets?keywords="GPCPDAY_3.2 GPCPMON_3.2"&page=1. I'm guessing that you will be interested in the daily dataset: https://doi.org/10.5067/MEASURES/GPCP/DATA305. This newer version has a higher spatial resolution, at 0.5 degrees.
I know that Brianna recently added the ability to query CMR for granules. You'll want to search for shortname GPCPDAY and version 3.2.

Thanks so much for investigating! This is so useful and interesting.
We can definitely point Pangeo Forge at the newest version of the dataset. However, NOAA NCEI is still distributing the older, corrupted version. Will you be reaching out directly to NOAA?

This is becoming a favorite use case of mine. For data provenance and responsibilities of data producers. As data producers we continuously update and potentially republish potentially erroneous data. One awesome feature in the future of pangeo-forge would be to understand and push those same updates. Data archives are more alive than one might think, so a single static moving of the data isn't sufficient. Not only for datasets still operational, but for corrections like these.
@rabernat we will contact the current data owners of that version.
Also would have been nice to quickly map the lineage of this dataset, without having 'insider' knowledge.
Thanks for catching this and shout out to @christine-e-smit for following up!

One more metadata comment: the original data's metadata includes a valid_range attribute. This is valid CF-1 metadata (https://cfconventions.org/cf-conventions/cf-conventions.html#missing-data) and would have been sufficient to screen out the values set to -99999. I still think that the missing_value should be consistent with what was actually put in the data, but the valid_range attribute does help.
$ ncdump -h gpcp_v01r03_daily_d20020907_c20170530.nc
netcdf gpcp_v01r03_daily_d20020907_c20170530 {
dimensions:
	latitude = 180 ;
	longitude = 360 ;
	time = 1 ;
	nv = 2 ;
variables:
	float latitude(latitude) ;
		latitude:long_name = "Latitude" ;
		latitude:standard_name = "latitude" ;
		latitude:units = "degrees_north" ;
		latitude:valid_range = -90.f, 90.f ;
		latitude:axis = "Y" ;
		latitude:bounds = "lat_bounds" ;
...
	float precip(time, latitude, longitude) ;
		precip:long_name = "NOAA Climate Data Record (CDR) of Daily GPCP Satellite-Gauge Combined Precipitation" ;
		precip:standard_name = "lwe_precipitation_rate" ;
		precip:units = "mm/day" ;
		precip:coordinates = "time latitude longitude" ;
		precip:valid_range = 0.f, 100.f ;
		precip:cell_methods = "area: mean time: mean" ;
		precip:missing_value = -9999.f ;

// global attributes:
		:Conventions = "CF-1.6, ACDD 1.3" ;
		:title = "Global Precipitation Climatatology Project (GPCP) Climate Data Record (CDR), Daily V1.3" ;
		:source = "/data1/GPCP_CDR/GPCP_Output/1DD//bin/200209/stfsg3.20020907.s" ;
...

Thanks so much for investigating! This is so useful and interesting.
We can definitely point Pangeo Forge at the newest version of the dataset. However, NOAA NCEI is still distributing the older, corrupted version. Will you be reaching out directly to NOAA?

We are going to contact Bob since he is the PI. We'll leave it up to him to contact NOAA NCEI.

Good news! We've gotten a response from the data curator, Guojun Gu, at the University of Maryland:

Hi Zhong,
I just did a quick check. Attached is the data file for 20020907 that is
archived in our machine. The missing value should be -99999, not -9999. I will
look into other files. Thanks!
Guojun

This ticket has been more fun for me than it really should be. It's been a while since I've embarked on trying to solve a data mystery and the solution to this one is extremely satisfying.
I agree with @briannapagan that no one should need insider knowledge to get to the bottom of an issue like this. When I've run into references to NASA earth science products I don't recognize (which is most of them!) before, I have used the feedback link at the top of the Earthdata Search website (https://search.earthdata.nasa.gov/search). Whoever is responsible for these tickets knows what products are archived where and knows who to contact. I went straight to the GES DISC help desk this time only because it looked as though the GES DISC had related datasets.
Better lineage would also help, including things like DOIs, which work really well to help find documentation on data products, even if they can't help you to figure out what all happened along the way.


Fetch refactor

Codecov Report

Merging #16 (900f313) into main (ce8e062) will increase coverage by 0.15%.
The diff coverage is 83.33%.

@@            Coverage Diff             @@
##             main      #16      +/-   ##
==========================================
+ Coverage   88.21%   88.36%   +0.15%
==========================================
  Files          22       25       +3
  Lines         976     1023      +47
  Branches      245      262      +17
==========================================
+ Hits          861      904      +43
- Misses        105      110       +5
+ Partials       10        9       -1

Impacted Files
Coverage Δ

src/validateList.ts
65.85% <65.85%> (ø)

src/createStores.ts
88.97% <92.00%> (+0.26%)
:arrow_up:

src/FetchList.vue
90.14% <100.00%> (-1.00%)
:arrow_down:

src/FetchSingle.vue
85.96% <100.00%> (+7.50%)
:arrow_up:

src/FetchSingleton.vue
94.73% <100.00%> (-1.56%)
:arrow_down:

src/ServerError.ts
100.00% <100.00%> (ø)

src/useFetchState.ts
100.00% <100.00%> (ø)

src/validateResponse.ts
100.00% <100.00%> (ø)

src/server/Server.ts
85.71% <0.00%> (+1.29%)
:arrow_up:

... and 1 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Adding a helper script for updating values in yaml

Can run as:
python3 update_yaml.py --input_path neat.yaml --keys name,description,node_path --values "kg-idg","kg-idg-xxxxx","./path-to-file"

Will recursively search for a provided key, but will not change keys found multiple times in the yaml OR keys nested within lists (as these may have an order we need to retain).

Looks good, but could we move the update_yaml.py script to the scripts directory?

Looks good, but could we move the update_yaml.py script to the scripts/ directory?

Done!


Plagiarism checks: Automatically select same file for both submissions in file comparison view

Is your feature request related to a problem?
When comparing two submissions in the plagiarism checks for programming exercises, the same file has to be selected for both submissions to actually assess the results of the JPlag runs. In the majority of cases, there is no point in comparing different files which can actually be misleading if different files contain similar content.
Describe the solution you'd like
Automatically select the same file (if exists) for both submissions and provide an option to disable this if necessary (e.g. a checkbox next to the confirm/deny and view split buttons)
Describe alternatives you've considered
No response
Additional context
No response

This would be really useful, otherwise the Report of JPlag is easier to use :)

But does it also select the files if you select another file ? This is a basic JPlag feature that might miss in Artemis

@dfuchss I misunderstood the original issue, yes that seems like a usable feature to have, I will assign it to myself.


wether to consider support "limit xx offset xx" statement

For now only support offset xx limit xx,
wether to consuder support limit xx offset xx?

The ANSI SQL syntax is:
OFFSET n [ROWS] FETCH NEXT m [ROWS] ONLY

(I am simplifying, there are some more variations there).
For user's convenience, and also because we already supported the non-standard LIMIT,
we decided to support the following:
OFFSET n LIMIT m

The LIMIT m OFFSET n syntax can be read both ways:

one can read this as "limit to m rows" then "discard first n rows", i.e. returning rows [m-n+1 ... m]
one can also expect this to behave as in some other databases, ie same as OFFSET n LIMIT m, ie returning rows [n+1 ... n+m]

To avoid this ambiguity, we decided not to support it.

Let me close the issue as answered. We can continue discussion here, but I also
invite you to join the #dev channel on our slack.


Added AdminSidebar Component

Description

What work was done?
Added AdminSidebar component to align with the AdminDashboard page.

Why was this work done?
The AdminSidebar routes the admin to all the admin pages

What feature / user story is it for?
Addresses the sidebar feature for this trello card

Any relevant links or images / screenshots

Wireframe for Admin Dashboard that displays AdminSidebar component on the left.

Co-authored-by: @kyle-naumann @ORiveraJr84
Initial rendering of Admin Dashboard that displays AdminSidebar component on the left.

Co-authored-by: @Regbmiller
Type of change

[x] New feature (non-breaking change which adds functionality)

Change Status

[x] Incomplete/work-in-progress, PR is for discussion/feedback

Has This Been Tested

[x] Yes

loom video

@Kishkuman13 Yes, this is a continuation of work after pull requests #375 and #376. #375 was just merged, so all good there. #376 I think should be merged before this one, that's the one on the AdminHeader.js file.


Update README.rst

Current wget install instructions result in a 404 response. Corrected path to kthresher.py

👍 Thanks for sending the fix, merging.


WIP - DO NOT MERGE Lb complete lifecycle new gh 675

Should it be Class<X> clazz?

It looks this way so that NamedContextFactory matches it. This has also puzzled me, but apparently that's the signature there.

Fixes gh-675.

Fixes gh-675.


⚠ Mobitopup has degraded performance

In c8cd0dc, Mobitopup (https://napcard.voz.tech) experienced degraded performance:

HTTP code: 200
Response time: 3300 ms

Resolved: Mobitopup performance has improved in 86e34a2.


Snapshot Manager Hangs and timeout after 300 seconds

I just deployed SnapshotManager using SAM with following configuration for hourly snapshot.
{  "snapshotIntervalHours": "1",  "targetResource": "rs-iam-test",  "namespace": "hourly",  "region": "us-east-1",  "snapshotRetentionDays": "7"}
CloudWatch event is triggering Lambda, but Lambda hangs for 5 mins and timeout after 300 seconds without any message in log.
Log messages:

START RequestId: 9fc47ef4-c6c7-4889-8270-16ad3fc87d54 Version: $LATEST
2019-03-19T08:34:38.491Z	9fc47ef4-c6c7-4889-8270-16ad3fc87d54	Configuration Provided:
{
"snapshotIntervalHours": "1",
"targetResource": "rs-iam-test",
"namespace": "hourly",
"region": "us-east-1",
"snapshotRetentionDays": "7"
}
2019-03-19T08:34:38.494Z	9fc47ef4-c6c7-4889-8270-16ad3fc87d54	Requesting
'hourly' Snapshots for rs-iam-test in interval (2019-03-19T07:34:38+00:00 - any)
END RequestId: 9fc47ef4-c6c7-4889-8270-16ad3fc87d54
REPORT RequestId: 9fc47ef4-c6c7-4889-8270-16ad3fc87d54	Duration: 300024.39 ms	Billed Duration: 300000 ms Memory Size: 192 MB	Max Memory Used: 38 MB
2019-03-19T08:32:49.163Z 9fc47ef4-c6c7-4889-8270-16ad3fc87d54 Task timed out after 300.02 seconds

I checked and updated IAM policy attached to the role for this Lambda. It has "redshift:*" on all resources. But still no luck.

When you see hang type behaviour on Lambda, it tends to be because Lambda cannot egress your VPC so that it can talk to the Redshift API's. Can you please consult https://github.com/awslabs/aws-lambda-redshift-loader/blob/master/Networking.md for more information on how to configure your Lambda function to work with AWS services, and then let us know if you still have an issue?

Amazon Redshift Snapshot Manager is now deprecated and you can use the native Redshift Snapshot Scheduler instead.


[1.12] Automated Bump DC/OS UI v2.26.17

High-level description
What features does this change enable? What bugs does this change fix?
Corresponding DC/OS tickets (required)
DCOS-57472
Related tickets (optional)

DCOS-57472
Checklist for component/package updates:
If you are changing components or packages in DC/OS (e.g. you are bumping the sha or ref of anything underneath packages), then in addition to the above please also include:

[ ] Change log from the last version integrated (this should be a link to commits for easy verification and review): example
[ ] Included a test which will fail if code is reverted but test is not. If there is no test please explain.
[ ] Test Results: [link to CI job test results for component]
[ ] Code Coverage (if available): [link to code coverage report]

Please update CHANGES.md file (Add or update DC/OS UI line in Notable Changes section of newest version).
* Updated DC/OS UI to [1.12+v2.26.17](https://github.com/dcos/dcos-ui/releases/tag/1.12+v2.26.17).

You can edit CHANGES.md directly here: https://github.com/mesosphere/dcos/edit/dcos-ui/bump/2019-08-12-07-58-17/1.12+v2.26.17/CHANGES.md?pr=%2Fdcos%2Fdcos%2Fpull%2F6058

@mesosphere-mergebot label [Ready for Review]

@pierrebeitz @GeorgiSTodorov CI is failing

@nLight, @jgehrcke from looking at the errors i'd assume it's no dcos-ui's fault. either i'm not allowed to or i'm too blind to find the rerun-button in teamcity.

Unfortunately, I can't see why it's failing.


changed the log from activateed to activated

When you submit a topology the log comes as activateed and deactivateed
. this is done because of the placeholder here and ed is added in
transitiontopologystate function

please sign the CLA since it is your first commit

HI @kramasamy  signed the CLA.
Vidit

ship it 👍


The occurrence of 429 error code responses has risen

I have recently upgraded my ADB version from 6.4 -> 12.2 LTS
which involves upgrading dependent libraries as well
i have upgraded azure cosmos spark version from azure_cosmosdb_spark_2_4_0_2_11_3_3_0_uber.jar to azure-cosmos-spark_3-3_2-12:4.20.0
Post that i am seeing increase in 429 request count in azure cosmos request metrics ?
Apart from that notebooks are running successfully, & the RU/S increase to 100% as some notebook start running as per triggers

@vipinsorot can you please provide some more details on your workload type? What operations are running and encountering these 429s?

@vipinsorot - closing this issue because of no activity, please re-open if the issue still persists.


Fixed #2186: Standardize tab theming

Description (required)
Changed the color of tabs of the bookmarks page, to standardize and look the same as categories page
Update: Also changed the search activity page
Fixes #2186: Night mode: Tabs
What changes did you make and why?
The only real changes happened in the activity_bookmarks.xml and activity_search.xml files. The other files(searchActivity, settingsActivity) show up as changes because because I was running a few tests but have since changed the code back to its original form, and when I ran the emulator (or updated sdk tools) android studio automatically applied a change to the build.gradle file from buildToolsVersion buildToolsVersion to buildToolsVersion '28.0.3' for some reason
Tests performed (required)
Testing done on LG V30 (direct) With API level 25
Testing done on nexus 5x emulator API level 28

Note: Please ensure that you have read CONTRIBUTING.md if this is your first pull request.

Codecov Report

Merging #2227 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #2227   +/-   ##
======================================
  Coverage    5.88%   5.88%
======================================
  Files         231     231
  Lines       11597   11597
  Branches     1081    1081
======================================
  Hits          683     683
  Misses      10856   10856
  Partials       58      58

Impacted Files
Coverage Δ

...fr/free/nrw/commons/settings/SettingsActivity.java
61.11% <ø> (ø)
:arrow_up:

...va/fr/free/nrw/commons/explore/SearchActivity.java
0% <ø> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b447612...8d0a4a6. Read the comment docs.

Sorry for the long wait on reviewing this.
Tested on Galaxy Nexus, API level 28:


XsLine.with(XSL)

One extra method.

@rultor release, tag is 0.1.27

@rultor release, tag is 0.1.27

@yegor256 OK, I will release it now. Please check the progress here

@rultor release, tag is 0.1.27

@yegor256 Done! FYI, the full log is here (took me 10min)

Job gh:cqfn/eo#301 is not assigned, can't get performer


There was an error processing your request: Could not go get the repository.

Hi,
Seems I'm not able to refresh the stats for my project.

https://goreportcard.com/report/github.com/dnaeon/gru

Could you please look into this one?
Thanks!

Hi! I can't go get the repo locally; it fails with the error:
# github.com/dnaeon/gru/utils
../../../dnaeon/gru/utils/file.go:26: undefined: user.Group

Could not go get the repository can happen for a number of reasons (there is a known bug), but this error is likely the issue in this particular case.

Hi,
Thanks for checking! Indeed that must be it, since I'm using golang tip. On a side note what is the golang version that gorepprtcard uses?
Thanks,
Marin

@dnaeon Our server is currently running go version go1.6.2 linux/amd64


Remove six in ci docker

Description
Attempt to fix lint issue in #19603 on master (as @leezu mentioned it also affect mater)
Checklist
Essentials

[ ] PR's title starts with a category (e.g. [BUGFIX], [MODEL], [TUTORIAL], [FEATURE], [DOC], etc)
[ ] Changes are complete (i.e. I finished coding on this PR)
[ ] All changes have test coverage
[ ] Code is well-documented

Changes

[ ] Feature1, tests, (and when applicable, API doc)
[ ] Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

applied the same changes, seeing the same build failures here https://github.com/apache/incubator-mxnet/pull/19174a

@mxnet-bot run ci [unix-cpu, website]


update the tikv community meeting

Signed-off-by: Wenting Liu<EMAIL_ADDRESS>
What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
What is changed and how it works?
Proposal: xxx
What's Changed:
How it Works:
Related changes

PR to update pingcap/docs/pingcap/docs-cn:
PR to update pingcap/tidb-ansible:
Need to cherry-pick to the release branch

Check List
Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression

Consumes more CPU
Consumes more MEM

Breaking backward compatibility

Release note

/merge


Support for Inf2 optimum class [WIP]

Adding support for inferntia.
Closes: #1343

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

Can you give this a first round of review.

Are there any additional comments, or can this be merged?

Didn't realize this was done! Yes, so long as unit tests pass, can be merged!

One last thing: could you add this model to the table of LM types and their names in the README, and also mention that it requires the optimum extra in the extras table?

Okay @haileyschoelkopf done that and improved the dependencies message on model loading.

Thank you, LGTM!
(still missing an entry in our table of LM integrations: https://github.com/EleutherAI/lm-evaluation-harness?tab=readme-ov-file#model-apis-and-inference-servers but I can add that in a later commit if desired, can't edit your PR)

Okay, updated the model name neuronx in the table and renamed all extras and hf_model to neuronx.


Add github action to run tests and linting on PR

I think this is complete with the 2 PRs. client #244 and server+adminui #246.
Maybe more lint:fix if we merge some PRs before this.
The current actions that run on pull_request_target and not very useful. Changes needed for this issue are done, but I didn't completely remove the useless ones.


When I use 8 A40 images to continue pre-training the XLSR-53 2B pre-training model, I always get an error message: Expected is_sm80 to be true, but got false. (could this error message be improved? If so, please report an enhancement request to pytorch) How can I solve this problem?

🐛 Bug

To Reproduce
Steps to reproduce the behavior (always include the command you ran):

Run cmd '....'
See error

Code sample

Expected behavior

Environment

fairseq Version (e.g., 1.0 or main):
PyTorch Version (e.g., 1.0)
OS (e.g., Linux):
How you installed fairseq (pip, source):
Build command you used (if compiling from source):
Python version:
CUDA/cuDNN version:
GPU models and configuration:
Any other relevant information:

Additional context

When I use 8 A40 images to continue pre-training the XLSR-53 2B pre-training model, I always get an error message: Expected is_sm80 to be true, but got false. (could this error message be improved? If so, please report an enhancement request to pytorch)
How can I solve this problem?


Rename "Image Reverse Search" to "Reverse Image Search" [U.S./U.K. English]

Technically, to be grammatically correct, it should be "Reverse Image Search".
Not really a major thing but thought I'd let you know. That's kind of my fault, I should have said something in the last issue I opened up. Everything else looks great! I'll test it across plenty of websites tomorrow.

No problem, shouldn't be a huge thing to change the name. I chose it because "Reverse Image Search" was already taken on AMO when I uploaded the add-on. It's also taken on the Opera Store so I can only change the display name on AMO and the name of the add-on itself plus the context menu.

Mkay! Well that's it. Everything is perfect. I didn't know it was already taken. My bad! :)


Filesystem Sharing

Hello. I would like to give some ideas and advices about filesystem sharing.
So, since we are using QEMU, lets see which options do we have:

VirtioFS. Looking very cool, seems to have really good performance, but works only on Linux hosts. It is very optimised for using in virtual machines, it even uses DAX (direct access) for files, so there's no need to copy files over network, they're just in the shared RAM between VM and host.
VirtFS (9P). I've tried to use it, but it's incredible slow. Really. Using just git status in shared directory with middle size project takes at least half a minute. I would rather just place files in VM and access them via some remote file access protocol and use vscode with remote access (sad, but they're proprietary). I think that it is so slow because it is sync. Whenever you read some file, do stat call etc. you have to wait for this operation to end.
Just sync filesystem state between VM and host.
Write custom FUSE driver with asynchronous protocol and multithreading support. Theoretically this can be more performant than 9P, but not sure.

My current plan:

Step 1: Use 9p over virtio serial: https://github.com/AkihiroSuda/lima/issues/1

Step 2: Implement vsock support in QEMU for Darwin, and switch away from virtio serial to vsock

Step 3: Implement virtiofs support in QEMU for Darwin

In parallel, we can consider supporting mutagen integration as well. IIUC it is used by Docker for Mac as well.
https://mutagen.io/

Step 3: Implement virtiofs support in QEMU for Darwin

Do you think this is possible? It requires to be able to share mmap'ped memory pages between host and guest. It this is possible, it would be cool.

Just so you know: There is a patch for Darwin 9p support developed within the Nixpkgs project. I have adapted it to current QEMU from a patchset originally by Keno Fischer.
The people over at Nixpkgs would certainly be interested in any effort to accelerate file sharing using VirtioFS, which is why I’m now subscribed here.

Thanks @mroi !

The patch seems proposed to the upstream in 2018, but what's the current status of upstreaming?
Is there / will there be a binary with the patch, that can be installed without nix?

@mroi Aside from 9p, do you know whether somebody is working on supporting vsock?
For reimplementing the even notification system that is currently implemented by running ssh -L /run/user/<UID>/lima-guestagent.sock:ga.sock.

I am preparing to propose the revised patch upstream. Have not had sufficient time to push this forward, but I hope to do this next week. It would certainly be good to have this upstream.
Regarding binaries: Nix has a binary cache, but the packages do not run standalone (they depend on other Nix packages). The patch however should apply to the vanilla QEMU sources, so you should be able to just recompile QEMU 6 with it.
I’m not aware of vsock developments. Unfortunately I’m not familiar with QEMU development internals at all. I just happened to have a need for 9p and worked on this patch.

Thanks!

The patch seems to have issues 😞 https://github.com/NixOS/nixpkgs/pull/122420#issuecomment-846365328

Until we can get virtio-9p-pci for macOS hosts in the QEMU upstream, I was planning to use 9P over virtserial, but it doesn't seem easy as expected 😞
diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index d9b0778..19f0f75 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -154,7 +154,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        // Parallel
        args = append(args, "-parallel", "none")

-       // Serial
+       // Legacy Serial
        serialSock := filepath.Join(cfg.InstanceDir, "serial.sock")
        if err := os.RemoveAll(serialSock); err != nil {
                return "", nil, err
@@ -167,7 +167,17 @@ func Cmdline(cfg Config) (string, []string, error) {
        args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server,nowait,logfile=%s", serialChardev, serialSock, serialLog))
        args = append(args, "-serial", "chardev:"+serialChardev)

-       // We also want to enable vsock and virtfs here, but QEMU does not support vsock and virtfs for macOS hosts
+       // vport for 9p
+       vportSock := filepath.Join(cfg.InstanceDir, "vport.sock")
+       if err := os.RemoveAll(vportSock); err != nil {
+               return "", nil, err
+       }
+       const vportChardev = "char-vport"
+       args = append(args, "-device", "virtio-serial")
+       args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server,nowait", vportChardev, vportSock))
+       args = append(args, "-device", fmt.Sprintf("virtserialport,chardev=%s,name=lima", vportChardev))
+
+       // TODO: use virtio-9p-pci when QEMU supports it for macOS hosts

        // QEMU process
        args = append(args, "-name", "lima-"+cfg.Name)

package main

import (
        "fmt"
        "os"
        "syscall"
)

func main() {
        if err := xmain(); err != nil {
                fmt.Fprintln(os.Stderr, err)
                os.Exit(1)
        }
}

func xmain() error {
        devPath := "/dev/vport5p1"
        mntPath := "/mnt/foo"

        devFd, err := syscall.Open(devPath, syscall.O_RDWR|syscall.O_NONBLOCK, 0600)
        if err != nil {
                return err
        }
        return syscall.Mount(
                "",
                mntPath,
                "9p",
                0,
                fmt.Sprintf("trans=fd,rfdno=%d,wfdno=%d", devFd, devFd),
        )
}

# go run main.go
invalid argument
exit status 1

# dmesg
...
[27353.823239] kernel write not supported for file /vport5p1 (pid: 2243 comm: kworker/3:1)

The error is happening because the vport device somehow wires up both write_iter and write: https://github.com/torvalds/linux/blob/v5.11/fs/read_write.c#L540-L545
I guess I have to inject some pipe as a "shim" fd, or use ssh instead of virtserial.

virtserial seems slower than ssh :(
virtserial
qemu flags: -device virtio-serial -chardev socket,id=foo,path=foo.sock,server,nowait -device virtserialport,chardev=foo.sock,name=foo
[host]$ socat file:urandom-1G unix-connect:foo.sock

[guest]$ time sha256sum /dev/virtio-ports/foo
real    0m10.022s

ssh
[host] $ cat urandom-1G | lima time sha256sum -
real    0m7.673s

virtserial seems slower than ssh

Given that we seem to be stuck with ssh, at least for now, did you check if selecting a different cipher makes any difference for throughput? It seems like we are using the default:
debug1: kex: server->client cipher<EMAIL_ADDRESS>MAC: <implicit> compression: none
debug1: kex: client->server cipher<EMAIL_ADDRESS>MAC: <implicit> compression: none

I remember reading that aes128-ctr was twice as fast as<EMAIL_ADDRESS>when running on a CPU with AES instructions, but not sure how much that affects throughput.
I'm somewhat confused though, as even when I add -c aes128-ctr to the ssh command, it still ends up using<EMAIL_ADDRESS>so I don't know how to configure it.

Given that we seem to be stuck with ssh, at least for now, did you check if selecting a different cipher makes any difference for throughput?

Haven't looked into. Help wanted :pray:

Btw QEMU Samba seems roughly two times faster than sshfs
[root@lima-archlinux ~]# time sha256sum /mnt/smb/ubuntu-21.04-desktop-amd64.iso
fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88  /mnt/smb/ubuntu-21.04-desktop-amd64.iso

real    0m15.578s
user    0m7.054s
sys     0m6.259s

[root@lima-archlinux ~]# time sha256sum /tmp/lima/ubuntu-21.04-desktop-amd64.iso
fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88  /tmp/lima/ubuntu-21.04-desktop-amd64.iso

real    0m29.707s
user    0m9.244s
sys     0m6.370s

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index 796c6ee..1524a24 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -193,7 +193,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        // CIDR is intentionally hardcoded to <IP_ADDRESS>/24, as each of QEMU has its own independent slirp network.
        // TODO: enable bridge (with sudo?)
        args = append(args, "-net", "nic,model=virtio")
-       args = append(args, "-net", fmt.Sprintf("user,net=<IP_ADDRESS>/24,hostfwd=tcp:<IP_ADDRESS>:%d-:22", y.SSH.LocalPort))
+       args = append(args, "-net", fmt.Sprintf("user,net=<IP_ADDRESS>/24,hostfwd=tcp:<IP_ADDRESS>:%d-:22,smb=/tmp/lima", y.SSH.LocalPort))

        // virtio-rng-pci acceralates starting up the OS, according to https://wiki.gentoo.org/wiki/QEMU/Options
        args = append(args, "-device", "virtio-rng-pci")

Needs https://github.com/Homebrew/homebrew-core/pull/80171 to be merged (which may take time)

Samba is now available on homebrew (https://github.com/Homebrew/homebrew-core/blob/master/Formula/samba.rb), so I'm planning to replace sshfs with samba soon.
Samba will be executed with slirp_add_exec: https://github.com/qemu/qemu/blob/4cc10cae64c51e17844dc4358481c393d7bf1ed4/net/slirp.c#L964
So Samba will not listen on any actual TCP port.
cc @jandubois

@AkihiroSuda That is awesome to hear. From the benchmarks i have seen so far, it is supposedly 2x faster with most workloads.
What would it take for us to support inotify events?

It would be interesting to see some benchmarks between sshfs and virtio-9p-pci and nfs and cifs/smb...
The virtfs authors seemed convinced:  https://www.kernel.org/doc/ols/2010/ols2010-pages-109-120.pdf

On Linux QEMU using virtfs is a lot faster than sshfs and even 9p. If anything is slowed down on macOS due to kernel lockdown/security don't take it as given for Linux users!
-–› Please always consider Linux users in mind since lima itself becomes more and more a tool for cross-platform development (macOS, Linux, WSL/Windows). Especially k8s/k3s is really nyce scenerio! Don't make your decisions based on macOS only! Thank you!
Docker for Desktop on Windows uses 9p (they switched away from SMB which they used before).

It's not Docker per se, but WSL2 that uses 9P for any WSL VM: https://devblogs.microsoft.com/commandline/a-deep-dive-into-how-wsl-allows-windows-to-access-linux-files/
I've been testing out WSL2 integration for other team members, and the filesystem performance is still not to a point where it's usable for large projects. For example, PHPStorm hangs and crashes trying to index a basic Drupal 9 codebase over the \\wsl$ share: https://github.com/drud/ddev/issues/3366
To date, mutagen (which is built into ddev) is giving our team the best experience. That work came about because Docker Desktop dropped their alpha mutagen support, and I think it would be really nice if support for it was at the lima (or colima) layer instead of at the next layer up.
I'm not sure it's worth the development effort to only get a relatively small performance improvement. If performance is any more than 20-50% slower than native IO, it's still too slow for most of our uses.

What would be needed for integrating mutagen ? It seems like it should work out-of-the-box, over the ssh connection ?
For docker-machine, the support was more "hands-on". The user had to run the scp and rsync and sshfs themselves...

Mutagen works, if you know your ssh and mutagen stuff.
I added an entry to ~/.ssh/config to disable host key check and specify the port, to make it easier to interact with rsync/mutagen:
HOST lima
Hostname localhost
Port 60022
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Then you can use a command like this to create a sync of the current working directory:
lima mkdir -p $PWD # create the needed folders
lima chmod 777 $PWD $(dirname $PWD) # mutagen will write a folder in the parent directory, so you need write access there too.
mutagen create \
  --sync-mode two-way-resolved \
  --ignore-vcs \
  --symlink-mode posix-raw \
  --default-file-mode 0666 \
  --default-directory-mode 0777 \
  --name "$(basename $PWD)" \
  $PWD lima:$PWD
mutagen monitor "$(basename $PWD)" # to watch your sync progress

Then stuff like lima nerdctl run -v $PWD:$PWD -w $PWD ubuntu ls will just run as expected. Just make sure your paths in the vm and on your machine are identical.

This could be a separate "mount" type, perhaps ? Similar to the current shorthand for setting up sshocker (reverse sshfs)
Didn't understand the part of the paths needing to the same on both sides, but will have to try it myself.

It could but... I feel like it won't be intuitive or stable. There are issues with very large amounts of files (multiple projects) write privileges, the amount of inotify watches on linux and there is conflict resolution etc.
I can manually enable sync's as needed. But i'm not too sure it will be a great fit as a solution in the config file.
But maybe i'm also not the average user ;)
Docker for mac had mutagen once as well https://github.com/docker/for-mac/issues/1592#issuecomment-678397258
There implementation was pretty good, it was integrated into the ui and showed sync progress etc. But you were on your own if it failed and it used a lot of resources when you synced larger directories.

At least you can use limactl show-ssh --format config default to set up the config.

Simple enough, though.
limactl show-ssh --format config default >>~/.ssh/config
mutagen sync create /tmp/test lima-default:/tmp/test
--------------------------------------------------------------------------------
Identifier: sync_WxCidKXaUaHGgkXXkRWZp5UBANGKpZlJ8JmJ0VgKepQ
Labels: None
Alpha:
	URL: /tmp/test
	Connection state: Connected
Beta:
	URL: lima-default:/tmp/test
	Connection state: Connected
Status: Watching for changes
--------------------------------------------------------------------------------

My two cents as the developer of Mutagen:
Mutagen works well if you have access to its full configuration and you can tweak it to your needs (e.g. via manual usage or Mutagen Compose) or if you're using it as part of some higher-level tooling that configures it in a way where it works for 99% of users automatically (e.g. the way ddev does).
It doesn't work well as a general purpose caching mechanism, and that was what really hampered the Docker for Mac integration (i.e. the simple on/off switch design and inability to see what's going on internally).
The inotify issue is something I'm working on addressing via use of fanotify and its enhancements in Linux 5.1+, but it's probably going to be restricted to very controlled environments (e.g. the sidecar container used by Mutagen Compose) because it requires CAP_SYS_ADMIN and CAP_DAC_READ_SEARCH and has somewhat odd path handling behavior with bind mounts.
Conflicts and handling of problematic content have been significantly improved in Mutagen v0.12, so hopefully that's less of an issue.
Anyway, I won't interrupt the discussion further, but feel free to reach out via email<EMAIL_ADDRESS>or Slack if you have any questions.

My idea was to leave it as opt-in, just wanted it easier to do...
I don't think it should be bundled, more treated as a tool (like sshfs)

Is there / will there be a binary with the patch, that can be installed without nix? For both Intel and ARM. (Would it be possible to crosscompile ARM target on an Intel host?)

@AkihiroSuda should we consider using go:embed to distribute static qemu-system-ARCH binaries?
Of course this would require a Workflow to maintain builds, but it would allow us to

Remove the dependency on having qemu installed
Apply the 9p support patch to Darwin builds

I don't think we should use go:embed, but we can consider putting patched QEMU binaries under /usr/local/share/lima, if it is acceptable for Homebrew/MacPorts/nix.

What is the difference between virtio-fs, virtio-9p, and virtio-blk? And which is simply called virtio?

"virtio-fs" is a whole new project (using vsock), while virtio-9p is the standard one used for "virtfs"

https://virtio-fs.gitlab.io/howto-qemu.html

https://wiki.qemu.org/Documentation/9psetup

virtio-fs borrows the FUSE request model and uses a daemon running on the host to service file access requests. There is also a direct memory extension that requires host kernel support called dax (Linux only). Using virtio-fs with dax allows the host and the guest to share cache and offers a big performance boost.

virtio-9p is just using 9p_net over virtio transport. It is a very old filesystem that has been stable in the kernel for a long time.

virtio-blk is not a file access layer but rather exposes a block device.

This seems like good news https://github.com/NixOS/nixpkgs/pull/122420#issuecomment-1022201942

Looks like there has been some progress/success with 9p and Qemu https://github.com/containers/podman/issues/8016#issuecomment-1034091359.

It is still enroute to be merged for 7.0, and still available as a patch for 6.x - so no real news from last year, except that more people are actually trying it now

After the 9p patch (https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg02090.html) gets merged we can switch the default from reverse sshfs to 9p, however, probably we will have to continue maintain reverse sshfs for EL8 family 😞
[root@lima-rocky ~]# grep -i 9p /boot/config-4.18.0-348.el8.0.2.x86_64
# CONFIG_NET_9P is not set

They are waiting for virtiofs support (or nfs over vsock). Waiting, waiting, waiting. It's been years already.

The 9p patch is now merged in the master branch of QEMU https://github.com/qemu/qemu/commit/f45cc81911adc7726e8a2801986b6998b91b816e

Note that it is the 9p for Darwin that is "new", the 9p for Linux has been there for ages

I added a "mountType" selector. Theoretically it could also host something like NFS.

Note that it is the 9p for Darwin that is "new", the 9p for Linux has been there for ages

Yeah, I guess most of us are interested in the Darwin version as we're trying to mount @ MacOS for replacing Docker Desktop.

The 9p patch is now also merged in brew, so hopefully that will be enough for most Mac users too

That's right -- the 9p patch has been backported to 6.2 in both brew and nixpkgs at this point and there's no reason why both will have any other issues making it to 7.0 regardless, so I'd consider it widely available.

The brew package is insanely large, since it includes some 30+ architectures. But that's another story...

Docker Desktop Edge has a VirtioFS option that is looking very good.

Docker Desktop Edge has a VirtioFS option that is looking very good.

VirtioFS is still unavailable in the macOS version of QEMU.
Apple's Virtualization.framework supports VirtioFS, but it is not the best fit for Lima, as it lacks support for firmware, qcow2, slirp, and a lot of features of QEMU. And yet it is proprietary.

Docker Desktop Edge has a VirtioFS option that is looking very good.

Looks like that dropped to stable! https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-460

Yes, it is available a an experimental feature and will be replace the gRPC FUSE implementation (hopefully). Instead of the hypervisor.framework it requires the new virtualization.framework which is available Big Sur. But VirtioFS in Docker Desktop for Mac is incredibly fast. With that feature the experience with developing inside containers is very comfortable. Would be very nice to have it in lima (perhaps optionally with additional libs / runtime options).

There is a new project which provides fuse via nfs, please follow these links for more info:
https://news.ycombinator.com/item?id=32726166
https://www.fuse-t.org/
IMHO, looks very promising. May be, it's would be very nice for lima ?

It has some similar licensing issues as macfuse, it seems ?

IMHO, looks very promising. May be, it's would be very nice for lima ?

It isn't open-source, and it isn't even free to use for commercial uses, so FUSE-T is not an option for Lima.
Also nfs is not really that simple to use on macOS: you need to be root to configure /etc/exports to secure the exported directories.

Hello, some interesting conversation in this thread. I was wondering, looking at this from a pragmatic lens, if there was an immediate solution (maybe the 9p patch getting folded in) while the longer term solution is in the works (which sounds like VirtioFS)? Myself, and others, would be thrilled to use Rancher Desktop on a regular basis, which is a downstream package that uses lima, but there's one thing stopping us: https://github.com/rancher-sandbox/rancher-desktop/issues/1209 (mounting volumes). Appreciate any thoughts or feedback.

Looks like 6 days ago Lima documented using virtiofs https://github.com/lima-vm/lima/commit/c18ae239b69a47db77436765b9b4861aaa0d595d

c18ae23

Yes, this is now available in v0.14.0-beta.0.

Any updates to this? Looking for a way to solve the missing inotify events in Colima and Lima.

Any updates to this? Looking for a way to solve the missing inotify events in docker-compose volume mounts in Colima and Lima.

inotify is discussed here:

https://github.com/lima-vm/lima/issues/615


airflow: make trino catalog non-mandatory.

Signed-off-by: Jakub Dardzinski<EMAIL_ADDRESS>Problem
Closes: #1554
Solution
Make trino catalog optional in trino extractor.
Checklist

[x] You've signed-off your work
[x] Your pull request title follows our guidelines
[ ] Your changes are accompanied by tests (if relevant)
[x] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
[ ] You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
[ ] You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

Codecov Report

Merging #1572 (603ed1e) into main (7fed782) will decrease coverage by 9.01%.
The diff coverage is 0.00%.

:mega: This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more
@@             Coverage Diff              @@
##               main    #1572      +/-   ##
============================================
- Coverage     82.77%   73.77%   -9.01%
============================================
  Files            88       48      -40
  Lines          3994     2341    -1653
  Branches         27        0      -27
============================================
- Hits           3306     1727    -1579
+ Misses          658      614      -44
+ Partials         30        0      -30

Impacted Files
Coverage Δ

.../openlineage/airflow/extractors/trino_extractor.py
64.00% <0.00%> (-28.00%)
:arrow_down:

...ntegration/airflow/openlineage/airflow/listener.py
0.00% <0.00%> (-73.27%)
:arrow_down:

...eage/airflow/extractors/redshift_data_extractor.py
39.47% <0.00%> (-50.00%)
:arrow_down:

...lineage/airflow/extractors/sagemaker_extractors.py
26.26% <0.00%> (-47.48%)
:arrow_down:

integration/airflow/openlineage/airflow/plugin.py
62.50% <0.00%> (-37.50%)
:arrow_down:

integration/airflow/openlineage/airflow/adapter.py
50.57% <0.00%> (-33.34%)
:arrow_down:

...w/openlineage/airflow/extractors/sftp_extractor.py
84.61% <0.00%> (-9.62%)
:arrow_down:

integration/airflow/openlineage/airflow/utils.py
67.89% <0.00%> (-9.04%)
:arrow_down:

integration/airflow/openlineage/airflow/facets.py
85.10% <0.00%> (-8.52%)
:arrow_down:

... and 42 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


Extra await inserted when accepting IntelliSense suggestion for property access on a promise-wrapped object

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.68-insider
OS Version: Windows 11

Steps to Reproduce:
In the following scenario, accepting the IntelliSense property access suggestion inserts an await without accounting for the existing await expression, resulting in redundant code. It would be nice if VS Code knew to reuse the existing await in this context.

Looks like a duplicate of https://github.com/microsoft/TypeScript/issues/48312


[1.7.10] Support craft/modtweakers.

Please add support craft/modtweakers for 1.7.10.

If anyone else wants to, then that would be awesome, but unfortunately, I really don't have the time to work on 1.7.10 anymore, and I barely have the time to continue updating for 1.10/11.2.


Outline missing corners

Version
v2000.2.9
What browsers are you seeing the problem on?
Chrome
What happened?
I created a simple rectangle with the same background as the game background (to look divisible), and added a white outline.
First, it just looked like rounded corners because the outline was too slim, but by increasing the size I noticed the corners are not rendered:

What's the expected behavior?
Filled corners. If possible, a cool feature would be allowing rounding these corners (by making arc).
Minimum reproducable code
const g = kaboom({ background: [19, 13, 33] });

		const w = g.width();
		const h = g.height();

		add([
			pos(w / 2, h / 2),
			rect(100, 100),
			outline(20, g.Color.fromArray([255, 255, 255])),
			area(),
			g.origin('center'),
			color(19, 13, 33)
		]);

Live demo
No response

Changed the default line join to rounded in f7c5189169e1184052cfb1bf849f094e496dd366


v1.9 backports 2021-06-02

[ ] #14639 -- test: Mark GKE CI pipeline as running Linux 4.19 (@pchaigno)

Commit 227860cbbccb (test: Mark GKE CI pipeline as running Linux 4.19) skipped: Already backported in #16224.
Commit 814a2f8ea2f7 (test: Fix fragment tracking test on GKE) backported in this PR (accidentally omitted from #16224).
Commit 0eb9b82aefff (test: Skip fragment tracking test on GKE) skipped: Already backported in #16224 (but reverted afterwards in 2dbe4c1c31d2).

[ ] #16176 -- contrib: simplify check-docker-images script (@aanm)
[ ] #16043 -- Support non-default Azure clouds (@ungureanuvladvictor)
First commit: Bump azure vendor from v44.2.0+incompatible instead of v50.2.0+incompatible as for master branch.
[x] #16137 -- docs: document the policy for backporting documentation changes (@qmonnet)
[ ] #16194 -- Make LRP restore test logic robust and optimized (@aditighag)
[ ] #16055 -- Add missing bpftool map dumps (@h3llix)

Once this PR is merged, you can update the PR labels via:
$ for pr in 14639 16176 16043 16137 16194 16055; do contrib/backporting/set-labels.py $pr done 1.9; done

test-backport-1.9

test-backport-1.9

I see some failures already.

k8s-1.17-kernel-5.4 (test-1.17-5.4) seems to be a flake, https://github.com/cilium/cilium/issues/16050
k8s-1.18-kernel-4.19 (test-1.18-4.19) seems to be a flake, https://github.com/cilium/cilium/issues/15255
gke-stable seems to be https://github.com/cilium/cilium/issues/16317, supposedly closed by https://github.com/cilium/cilium/pull/16333, which contains the revert from @pchaigno's commit. Paul, it seems that 227860c does not work on v1.9, even with 814a2f8 applied? (Fragment testing should be skipped on GKE anyway as per 0eb9b82, so I don't think 814a2f8 would solve the issue?)

* `gke-stable` seems to be #16317, supposedly closed by #16333, which contains the revert from @pchaigno's commit. Paul, it seems that [227860c](https://github.com/cilium/cilium/commit/227860cbbccb06cb0e9d904be73f3be7b2ba2cbf) does not work on v1.9, even with [814a2f8](https://github.com/cilium/cilium/commit/814a2f8ea2f70133453565776dca5480b342072e) applied? (Fragment testing should be skipped on GKE anyway as per [0eb9b82](https://github.com/cilium/cilium/commit/0eb9b82aefff44ff59b928113ffbc8cfbb0aa0d8), so I don't think [814a2f8](https://github.com/cilium/cilium/commit/814a2f8ea2f70133453565776dca5480b342072e) would solve the issue?)

That test doesn't fail on master because we skip it on master. Let's do the same here. Could you edit the commit to skip Tests NodePort (kube-proxy) when running on GKE and add a note in the commit to explain? The corresponding master code is here: https://github.com/cilium/cilium/blob/7a4184f1195c0dd81a84cd3b265de19fb0f0fbb8/test/k8sT/Services.go#L2059

I updated according to Paul's suggestion (and rebased on top of current v1.9). His first commit in this PR is the only one which changed.
git diff qmonnet/pr/v1.9-backport-2021-06-02 -- test
diff --git a/test/helpers/utils.go b/test/helpers/utils.go
index c28e097b7885..8f2f53f47c90 100644
--- a/test/helpers/utils.go
+++ b/test/helpers/utils.go
@@ -561,6 +561,13 @@ func DoesNotRunOnEKS() bool {
 	return !RunsOnEKS()
 }

+// RunsWithKubeProxyReplacement returns true if the kernel supports our
+// kube-proxy replacement. Note that kube-proxy may still be running
+// alongside Cilium.
+func RunsWithKubeProxyReplacement() bool {
+	return RunsOnGKE() || RunsOnNetNextKernel() || RunsOn419Kernel() || RunsOn54Kernel()
+}
+
 // DoesNotHaveHosts returns a function which returns true if a CI job
 // has less VMs than the given count.
 func DoesNotHaveHosts(count int) func() bool {
diff --git a/test/k8sT/Services.go b/test/k8sT/Services.go
index da4562222811..6fcba71a7f72 100644
--- a/test/k8sT/Services.go
+++ b/test/k8sT/Services.go
@@ -1563,7 +1563,7 @@ var _ = SkipDescribeIf(helpers.RunsOn54Kernel, "K8sServicesTest", func() {
 			}
 		}

-		SkipContextIf(helpers.RunsWithoutKubeProxy, "Tests NodePort (kube-proxy)", func() {
+		SkipContextIf(helpers.RunsWithKubeProxyReplacement, "Tests NodePort (kube-proxy)", func() {
 			SkipItIf(helpers.DoesNotRunOn419OrLaterKernel, "with IPSec and externalTrafficPolicy=Local", func() {
 				deploymentManager.SetKubectl(kubectl)
 				deploymentManager.Deploy(helpers.CiliumNamespace, IPSecSecret)

test-backport-1.9

Failure on k8s-1.12-kernel-netnext (test-1.12-netnext) is known as https://github.com/cilium/cilium/issues/16122
Failure on k8s-1.17-kernel-5.4 (test-1.17-5.4) is known as https://github.com/cilium/cilium/issues/16050


Bug Fix: create userDataDir if it doesn't exist

This fixes the issue where the userDataDir is not found when attempting to launch the Edge instance.  This results in a failure and aborts the launch process.  This only repros when launching the browser instance from our extension (since the failure is from puppeteer and we use puppeteer to launch) and when the user's userDataDir does not exist.
(e.g. C:\Users\{username}\AppData\Local\Temp\vscode-edge-devtools-userdatadir_9222)
close #861

I also met this problem！！！


Updated RPIP-49

Aimed for a slightly more formal overview tone.
Aimed to explain what the key components are and their general benefits.
Added excluded components back in limited form.
Fixed minor typo in RPIP-44.
Switched RPIP-49 to living informational to better reflect that it'll evolve.
Would also like to add more links to further links section, but I've lost all the links to @orangesamus's spreadsheets.
Updated descriptions in the content RPIPs, generally aiming for more clarity.
Current Status section is TBD. Feels like a reasonable chunk of work, and I need to dig around again to see where we are.

Note I worked from latest official main as a base, then merged in Samus's open PR. Merge official main and take Samus's and that should clear out the cruft changes from the github PR UI (I think).

This too:


Sampling per partitions

This PR implements sub-sampling of partitions for utility analysis.
It works the following:
1.UtilityAnalysisEngine.aggregate has sampling_probability
2. Utility analysis ContributionBounder can perform sub-sampling of partitions using ValueSampler if sampling probability < 1.0
3. ValueSampler computes hash of partition key and if the hash (which is assumed to return uniform values) is less than sampling_rate*max_hash_value, the partition_key is kept.

Thanks for review!


Show liquidity page for not logged in users

Closes: #2239
Closes: #2257
Closes: #2258
Closes: #2259
Show liquidity page for users that did not connect their wallet yet.

@michalsmiarowski could you please resolve conflicts with a master branch?

@michalsmiarowski could you please resolve conflicts with a master branch?

Ready for another review. I also resolved the conflicts with the master branch.

Ready for another review. I also resolved the conflicts with the master branch.


Created pull request template

This PR adds a template for creating new pull requests, resolving issue #7.
The following changes were made in this PR:

Created a .github folder in the repository's root.
Within this folder, created a subdirectory, "PULL_REQUEST_TEMPLATE" to organize the related files.
Created a PULL_REQUEST_TEMPLATE.md file which contains the pull request template.

@CBID2 I'll effect these suggestions immediately. Also, thank you for the tip.

@CBID2  I've effected the suggested changes in my PR.

Did you use any template to create this template?

Did you use any template to create this template?

Yes, I did.

Let's not use a template here. Chances are, it's too long and some of it is not relevant for us

Let's not use a template here. Chances are, it's too long and some of it is not relevant for us

What do you recommend as an alternative @ykdojo?

It'll be better if we make it from scratch and expand from there.

So a major issue with this current  template is the length ?

Yes that is a major thing with it

Okay. I'll redraft it to suit the current stage of the project. Thank you for your feedback.

Hi @ykdojo, I have made corrections to the PR as requested. Please review.

I'm still not sure how this would add value to this project. It includes a mention of a style guide, for example, which we don't have.

It should be built from scratch without using a template.

No activity here in two weeks. I'll close it if @BlackBond06 has no updates.


fix:(examples/nextjs) removed unused property that caused confusion

:tada: This PR is included in version 3.12.1 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


read-only mode allows paste

From chrome web store, user Amanpreet Singh(TECH - BLR):
"You can paste content when in Read only mode by accident and there is no way to remove it.
I am using it in conjunction with clipboard recorder like alfred. When you select the content to search first of all custom search bar goes away and things gets pasted inside the Json View.
Also custom Search Bar should not go away while i am selection an item to paste. Default chrome search bar has this behaviour. I use CMD+CTRL+V to bring paste menu due to which Viewers custom search bar goes away."

This was fixed here but a fix was provided in CodeMirror 5.6.x https://github.com/codemirror/CodeMirror/commit/0a54ddbc70390dcad5bf3afc408adbf24675294e
just for future reference


rqt_console: "Fit Columns" button cannot be easily undone / has a counter-intuitive name

From @jbohren on November 4, 2013 14:23
In rqt_console, there is a "Fit Columns" button, which expands all the columns so that their contents isn't truncated. I propose that "Fit Columns" should be renamed to a toggle button "Expand Columns" and an additional, mutually-exclusive toggle button should be added which causes the columns to fit within the window as it gets resized.
Copied from original issue: ros-visualization/rqt_common_plugins#195

That should be fairly easy to integrate. Can you please explain how you would like the columns sizes to be allocated within the window size?

From @ablasdel on July 3, 2014 19:1
@jbohren If we can lock down the important behaviors that are missing this should be pretty trivial for me to add them.
I am envisioning replacing the fit columns button with a general button that brings up a list with a few reasonable options  to adjust the column state.


Library changes my color scheme

When adding the library to my project, the colors of my buttons and fab's change without any OTHER changes.
To Reproduce
Add this to your project:
implementation ('com.schibstedspain.android:leku:6.2.0') {
    exclude group: 'com.google.auto.value'
}

Expected behavior
Colors should not change
Info Required
The color is changed for all activities, not only the place picker one.
Screenshots
Before:

After:

Smartphone:

Device: Samsung Galaxy S9

Additional context
My styles.xml:
<style name="LevionAppTheme.Dark" parent="Theme.MaterialComponents.NoActionBar">
    <!--item name="android:windowFullscreen">true</item-->
    <item name="windowActionModeOverlay">true</item>
    <item<EMAIL_ADDRESS>    <item<EMAIL_ADDRESS>    <item<EMAIL_ADDRESS>    <item<EMAIL_ADDRESS>    <item<EMAIL_ADDRESS>    <item name="android:textColorPrimaryInverse">@color/textColorPrimary</item>
    <item name="android:textColorSecondaryInverse">@color/textColorSecondary</item>
[...]

Manifest:
<application
    android:name=".MyApplication"
    android:allowBackup="true"
    android:icon="@mipmap/ic_launcher"
    android:label="@string/my_app_name"
    android:theme="@style/LevionAppTheme"
    tools:replace="android:theme,android:label"
    android:largeHeap="true"
    >

<activity
    android:name="com.schibstedspain.leku.LocationPickerActivity"
    android:configChanges="screenSize|orientation|keyboardHidden">
    <meta-data
        android:name="android.support.PARENT_ACTIVITY"
        android:value=".MainAppActivity" />
</activity>

Hi @DennyWeinberg
This should not happen since the library use the resource prefix to avoid this kind of issues.
Looking at the code you shared the only thing that could avoid this to happen is the line in the manifest:
tools:replace="android:theme,android:label"
You are telling to the manifest merger to override the theme in case it's duplicated. Try to do the same on the activity you are experiencing this issue by adding the same "android:theme".
However, I'll try to reproduce the issue.

Hello,
I need that because of that dumb net.rdrei.android.dirchooser library. (https://github.com/passy/Android-DirectoryChooser/issues/85)
I have to find a solution. I already know that it's not your library who causes the problem, because I updated some other libraries and now I have the same issue in my app without using your lib.

@DennyWeinberg
Since is not related to this library I have not much to say. I could help you but It would be helpful to have an example that reproduces the problem. Please could you provide a github repository reproducing the error? And I'll try to help. Otherwise, I'll close this issue.

Ok. Thank you! I will contact you when I am able to reproduce it in a small project.


feat: updated game board to expand to half the page

Fixes #12

Great job @Ayoazeez26
However, the pieces on the board does not well positioned in their various homes and when they start moving, they don't move along the tile nodes of the board.
Please can you fix that and create a PR ASAP?

Sure @vibenedict I’ll fix and revert


withPWA precaches on every route, but I want it to do that only on specific routes

The main issue I'm trying to figure out right now is I noticed workbox/the service worker that withPWA sets up, starts precaching everything I specified but on EVERY route, such as our login page. But I would like to restrict this to just precache when on specific routes like /test or /other so it doesn't bog down my entire website for no reason.
Is there a way to do this. Here is my current config
await import("./src/env.mjs");
import { nanoid } from "nanoid";
import i18n from "./next-i18next.config.js";
import withPWA from "next-pwa";

const buildId = nanoid();

const pwa = withPWA({
  dest: "public",
  scope: "/test",
  additionalManifestEntries: [
    { url: "/logo.svg", revision: "1" },

    { url: "/test", revision: "1" },
    { url: "/other", revision: "1" },

    { url: "/en/test", revision: "1" },
    { url: "/en/other", revision: "1" },
    { url: `_next/data/${buildId}/en/test.json`, revision: "1" },
    { url: `_next/data/${buildId}/en/other.json`, revision: "1" },

    { url: "/es/test", revision: "1" },
    { url: "/es/other", revision: "1" },
    { url: `_next/data/${buildId}/es/test.json`, revision: "1" },
    { url: `_next/data/${buildId}/es/other.json`, revision: "1" },

    { url: `_next/data/${buildId}/en.json`, revision: "1" },
    { url: `_next/data/${buildId}/es.json`, revision: "1" },

    // ... other routes
  ],
});

/** @type {import("next").NextConfig} */
const config = pwa({
  generateBuildId: () => buildId,
  reactStrictMode: true,

  /**
   * If you have `experimental: { appDir: true }` set, then you must comment the below `i18n` config
   * out.
   *
   * @see https://github.com/vercel/next.js/issues/41980
   */
  i18n: i18n.i18n,
});

export default config;

You can add buildExcludes: [/\/_next\/static\/.*\.js/] to skip precaching every JS files that Next.js emits for its pages,  but I'm not aware of any way to still precache certain pages' after opting out of that. If we could read app-build-manifest.json and build-manifest.json in buildExcludes, that would have been a different story, but here we are.
This whole precaching doesn't block your page from functioning as it is done in the background, by the way.


Link legacy staking page to Gitbook guide on how to unstake LP

In v1, Ohmies could stake their ohm/dai LP. We have since deprecated that feature and no longer grant rewards. Rather than support that feature in the UI we have a a GitBook Link:
https://docs.olympusdao.finance/using-the-website/unstaking_lp
Needs design input.

Design proposal
https://www.figma.com/file/Yv3MynoOfFbFpd641XTl0z/stake-bond?node-id=669%3A19751


feat: download kusion modules with current os arch filter

What type of PR is this?

What this PR does / why we need it:
Which issue(s) this PR fixes:

Fixes #
Special notes for your reviewer:
Does this PR introduce a user-facing change?

Additional documentation e.g., design docs, usage docs, etc.:

Pull Request Test Coverage Report for Build<PHONE_NUMBER>
Details

0 of 15 (0.0%)  changed or added relevant lines in 2 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.03%) to 54.831%

Changes Missing Coverage
Covered Lines
Changed/Added Lines
%

pkg/engine/api/generate/generator/generator.go
0
2
0.0%

pkg/engine/api/generate/run/run.go
0
13
0.0%

Totals

Change from base Build<PHONE_NUMBER>:
-0.03%

Covered Lines:
8830

Relevant Lines:
16104

💛  - Coveralls


Ci fixes

Replying comment that was deleted by force push.
I added a reference to cabal bug, and removed dead code.
Commit should be reverted once new cabal version will be released.