[WIP] Travis: aom or dav1d decode tests, pick one

[x] For now choose decode_test_dav1d, it should be faster.
[ ] Ensure that the skipped variant can be built.

I think we need a discussion on this. I was asked specifically to run both, I believe by @YaLTeR (although I'm bad at remembering who tells me things).

The purpose of the first patch is to gauge the impact of running both. If it is significant, then we may need to find the balance between CI responsiveness and avoiding silent breakage.

Coverage remained the same at 70.678% when pulling 56014d04e6b3e6551363bc0f08cc4a532e7712ff on barrbrain:aom-dav1d-pick-one into e44bf1cb93ceba16e6a845e27f1a1a9fbac80712 on xiph:master.

Another thing we could try is run aom and dav1d ignored tests in separate jobs. This way they should run in parallel, potentially giving a speed-up?

There's a little bit of waste from having to build rav1e twice. But hopefully not too bad, especially if the cache is used. That would be faster (based on wall time).

Appveyor had a timeout.


Edit Task Templates Disappear/Gone

Once i was changing my password where my user is an admin role, the edit pencil icon on all Tasks Templates was not available anymore.
Is there anything i can do ?
With a default Admin user where i havent change the password.

With my Admin user where i have change the password.

Are there any updates / work arounds to this?

v2.10.35-a71a3c56-1731092948


Seeing image tinted green when resized.

When viewing the raw image without the proxy the image is displayed fine, when the image is resized it comes back with a green tint as though something has gone wrong with the image channels. Wonder if you have seen this before and whether there may be something misconfigured on our side that would cause this?
Raw Image: https://cl.ly/0e472t1s2T3v
Proxied: https://cl.ly/1N2925313c2V

Weird.  No, I've certainly never seen anything like this before.
Here's a quick attempt at uploading your raw image to my server and proxying it, which looks fine:
https://willnorris.com/api/imageproxy/800/https://willnorris.com/tmp/test-image.jpg

I've tried a couple of times and am not able to reproduce this.  Is this still happening for you?  Are you seeing it with just this one image, or others as well?

closing old issue with no response.


Repo code giving errors after download

Hi,
Can you please give the error free code of this project. And I need to use laptop's fingerprint scanner's data to save and verify. Please can you suggest how can I achieve this.
My requirements are:

Save an employee's finger data (taken from system's built in fingerprint scanner) along with other data to database
access and verify that employee
take picture (snapshot) of employee's with camera

This is not an issue

But I can't build/run this repo code on my machine it gives errors. Can you help in this?

Possibly. What are the errors?


Add 'Advanced Config' Button and Menu

Developers will want to tweek the way the demo wallet operates in order to test all edge cases on their anchor.
For example, Settle asked us to add a on_change_callback field to the existing demo client today. The demo wallet should also be able to do that. Note that this includes updating the URL so the config is completely sharable.

@JakeUrban Is this still a feature request in the revamped Demo Wallet? If so, where would this be configured @lydiat

I think the plan was to implement this in V2. The idea is that there are optional features provided in the SEPs, and the anchor using our demo wallet will likely want to test the implementation of those optional features by explicitly enabling them.
We could take the approach of defaulting to using this optional features, but then the anchor wouldn't be able to test flows where those optional features are not requested/enabled.
For example, anchors can optionally create claimable balances for users requesting a SEP-24 deposit, but only does so if the wallet uses the claimable_balance_supported: "true" flag in the deposit request. Our current demo wallet passes this parameter, but this means anchors implementing this feature cannot test for the case where the wallet doesn't send the parameter.
If we had an optional config menu, the user could toggle whether or not this claimable_balance_supported flag was used, enabling testing of both flows.


Example Request About BLE Server (IDFGH-4427)

Hi there,
I am trying to implement gatt server example on IDF. But it is too hard for me. I learn, i have to improve some C skills first. I want to create an app that communicate with mobile phone over BLE. I know and I used kolbans libraries and its time to deep dive to ESP32. So i decided learn IDF.
I checked gatt server example walkthrough documentations and changed some lines on the code and i saw changes. But i think i need simple explanation and code snippets for basic structure.
Like Using custom UUID's, creating services, characteristics, read-write-notify and interact with client(mobile app or pc etc).
Can you suggest any source about that?

Each API is annotated with instructions on how to use it, and you can refer to the Bluetooth protocol if you need to understand the corresponding concepts.


build-icons doesn't work with entypo

Description
Trying to run yarn run build-entypo fails with as the zip-file to be downloaded results in a 404: https://dl.dropboxusercontent.com/u/4339492/entypo.zip

This was fixed, now using npm package


[netcore][11.qnamaker] Missing files and Readme has wrong instruction

Sample information

Sample type: [\samples]
Sample language: [dotnetcore]
Sample name: 11.qnamaker

Describe the bug

The bot.recipe is missing the resource qna.

BotConfiguration.bot is missing the qna service

Step 2 from Prerequisites in readme.md points to import the wrong archive
File smartLightFAQ.tsv missing from directory \11.qnamaker\CognitiveModels\
File 34.qna missing from directory \11.qnamaker\deploymentScripts\msbotClone

To Reproduce
Steps to reproduce the behavior:

Follow the Prerequisites steps from readme.md up to step 2
Import sample.qna to QnA portal
The following message will appear: The file format is not supported for sample.qna. Incorrect file format. Import operation expects a .tsv file with question, answer, source and metadata fields.

Expected behavior
The steps in prerequisites should say to import a .tsv file. Said file should be present in the directory \11.qnamaker\CognitiveModels\.
After importing the correct file, the Knowledge Base should be updated with the content of the file.
Additionally, following the steps in the readme.md the BotConfiguration.bot should have the corresponding properties to update with the QnA information.
Screenshots
This is the error that appears when trying to import the .qna to QnA portal.

The file on the left shows the content of the bot.recipe of sample # 11, the file on the right shows the how the bot.recipe should have (taken from the same sample but for JavaScript).

Closed due to Microsoft#628


Warning: Each child in a list should have a unique "key" prop

Hello,
Thank you for the great plugin! I tested your examples in StoryBook and installed it locally. However, I encountered a problem in React.
Every time I use the component, the following warning appears:
Warning: Each child in a list should have a unique "key" prop.

Steps to reproduce:
Install the plugin locally.
Use the component in a React application.
Open the browser console and see the warning.

Expected behavior:
There should be no warnings in the console.
Additional information:
It would be great if you could add unique keys for each list item.
I have attached a screenshot of the problem:

Thank you!

Hi @nurlanmamedov , thanks for the issue!
I just merged a fix, the warning should be gone now.
Let me know if you run into any issues 🙂

Hi @nurlanmamedov , thanks for the issue! I just merged a fix, the warning should be gone now. Let me know if you run into any issues 🙂

Cool, thanks 🙂


Visual Studio crash (FailFast) when using new style fsproj files

I converted a couple of my project files over to the new fsproj format, and now I'm getting this crash every time I open Visual Studio:
Application: devenv.exe
Framework Version: v4.0.30319
Description: The application requested process termination through System.Environment.FailFast(string message).
Message: System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.CodeAnalysis.Editor.Implementation.Structure.AbstractStructureTaggerProvider`1.<ProduceTagsAsync>d__6.MoveNext()
Stack:
   at System.Environment.FailFast(System.String, System.Exception)
   at Microsoft.CodeAnalysis.FailFast.OnFatalException(System.Exception)
   at Microsoft.CodeAnalysis.ErrorReporting.FatalError.Report(System.Exception, System.Action`1<System.Exception>)
   at Microsoft.CodeAnalysis.ErrorReporting.FatalError.ReportUnlessCanceled(System.Exception)
   at Microsoft.CodeAnalysis.Editor.Implementation.Structure.AbstractStructureTaggerProvider`1+<ProduceTagsAsync>d__6[[System.__Canon, mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at Microsoft.CodeAnalysis.Editor.Implementation.Structure.AbstractStructureTaggerProvider`1+<ProduceTagsAsync>d__6[[System.__Canon, mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].MoveNext()
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.InvokeMoveNext(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
   at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(System.Action, Boolean, System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.FinishContinuations()
   at System.Threading.Tasks.Task.FinishStageThree()
   at System.Threading.Tasks.Task`1[[System.__Canon, mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
   at System.Threading.Tasks.TaskCompletionSource`1[[System.__Canon, mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].TrySetResult(System.__Canon)
   at<EMAIL_ADDRESS>mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Invoke(System.Exception)
   at Microsoft.FSharp.Control.CancellationTokenOps+StartWithContinuations@964-1.Invoke(System.Runtime.ExceptionServices.ExceptionDispatchInfo)
   at <StartupCode$FSharp-Core>.$Control.loop@124-51(Microsoft.FSharp.Control.Trampoline, Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.FakeUnitValue>)
   at Microsoft.FSharp.Control.Trampoline.ExecuteAction(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.FakeUnitValue>)
   at Microsoft.FSharp.Control.TrampolineHolder.Protect(Microsoft.FSharp.Core.FSharpFunc`2<Microsoft.FSharp.Core.Unit,Microsoft.FSharp.Control.FakeUnitValue>)
   at<EMAIL_ADDRESS>mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]](Boolean, Microsoft.FSharp.Control.AsyncParams`1<System.__Canon>, System.Threading.Tasks.Task`1<System.__Canon>)
   at<EMAIL_ADDRESS>mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Invoke(System.Threading.Tasks.Task`1<System.__Canon>)
   at System.Threading.Tasks.ContinuationTaskFromResultTask`1[[System.__Canon, mscorlib, Version=<IP_ADDRESS>, Culture=neutral, PublicKeyToken=b77a5c561934e089]].InnerInvoke()
   at System.Threading.Tasks.Task.Execute()
   at System.Threading.Tasks.Task.ExecutionContextCallback(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef)
   at System.Threading.Tasks.Task.ExecuteEntry(Boolean)
   at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

I'm using a vsix compiled from the latest commit (d39cba7b0fe4247f05f4065927dc4868ec34cb6a), and this happens both on VS 15.3 and VS 15.4 preview 4.

I don’t think Visual F# Tools support new style projects (yet?)

The tools in 15.3 and 15.4 do not have proper support for .NET Core SDK-based projects.  This is something targeted for 15.5, which there aren't public builds of yet.
That said, the stack trace there looks problematic, and we haven't seen that happen before.  BlockStructure was causing a nullreference exception in the Microsoft.CodeAnalysis layer in 15.3 and 15.4.  @KevinRansom @Pilchie for FYI

I'm on 15.4 now, and I've realized that this happens only when opening VS with a file from the new style fsproj already open.  After VS successfully loads, these files can then be viewed without any issues.
I haven't been able to reproduce this on a simple test solution, so it may be something specific to my real-world solution - I'll test again and update this issue once the 15.5 preview starts up.

This is still happening on VS 15.5 preview 1 with the vsix compiled from b714ecb83648de5d145c49c9939f02766c7280a8.
With the vsix included in VS 15.5 preview 1 there is no crash, but I don't get any semantic highlighting in the files in my new style fsproj (I do have working expand/collapse regions though).

cc @KevinRansom

@dhwed Can you provide a project or solution where you see this?  I just tested this with the SAFE Bookstore and it loaded/restored packages and had features light up without issue.

Looks like this is happening because I have a Common.props file so that things like the warning level can be defined in one place and then be consistent between all of my projects.
So, this can be reproduced with a dotnet new console -lang F# project with two minor modifications:
Add a new Common.props file with this content (my file actually has a <PropertyGroup>, but it isn't necessary to reproduce the crash):
<Project></Project>

and then add this line to the fsproj:
<Import Project="Common.props" />

Open the fsproj in VS, compile, then open the Project.fs file.  Then close and reopen VS, and it crashes.

This isn't happening any more after @KevinRansom's p2p references work, so I'm going to close this.


Decide on consistent behaviour of math ops

There has been some disagreement about the level of checks performed by add/subtract and multiply/divide.  Up to Iris 1.12, the behaviour was not consistent between the two groups of math ops (add/subtract had more strict coordinate matching checks than multiply/divide).
This PR changes the behaviour of multiply/divide to be consistent with add/subtract: https://github.com/SciTools/iris/pull/2498
This was merged and released in Iris 1.13, but @rcomer has raised concerns that the checks may be too robust, and may need some deeper thought before we decide on which consistent behaviour math ops should follow.
See here for use case

Thanks @corinnebosley.  I'm very glad that this issue is getting attention as subtraction between cubes has been my biggest source of frustration as an Iris user.  I absolutely agree that the behaviour should be consistent across the binary operations.
I think examples where the checks fail fall into two categories:

A coordinate on cube has no associated coordinate on other.  Examples of this are quite straight-forward.  E.g. using a scalar cube to convert a variable (as linked in the OP):

if cube.name() == 'geopotential':
    # Convert ERA-I geopotential to geopotential_height.
    gravity = iris.cube.Cube(9.80665, units='m s-2')
    cube = cube / gravity

Another example might be to compare forecast data from an ensemble (realization x time x latitude x longitude) with observed data (time x latitude x longitude).
Here is an example which fails because an auxcoord is missing, even though the dimcoords are identical:
fname = iris.sample_data_path('uk_hires.pp')
cube = iris.load_cube(fname, 'air_potential_temperature')
other = cube.copy()

iris.coord_categorisation.add_hour(cube, 'time')
cube - other

Coordinates that look similar (i.e. have at least the same name) but are "ungroupable".

This may happen simply because the coordinates have a different var_name, which will happen quite often if you're comparing data from different sources.  So I definitely think we should relax the requirements as far as the coordinate metadata is concerned.
Bounds also need to match up, which again may not happen if your data come from different sources or have been processed differently, or if you e.g. want to know the difference between some instantaneous daily values and their respective daily means.
fname = iris.sample_data_path('air_temp.pp')
cube = iris.load_cube(fname)
other = cube.copy()

cube.coord('latitude').guess_bounds()
cube - other

If I've understood from #2498, a check on the points was required by a user.  I have a few thoughts on this so I'll put them in a separate post...

Thanks @rcomer!  Got some really good use cases there, really appreciate your time and effort.  I will ping you when we get anywhere with this, which should be either this week or next.

Here are some examples where coordinate points might legitimately differ:

I think it is reasonable for a user to want to do something like cube[n:] - cube[:-n].

Here is a recipe for creating 2 coords which represent identical times, but have different points and units:

def time_coord(datetimes, unitstr='hours since 1970-01-01 00:00:00',
               calendar='gregorian', standard_name='time'):
    """
    Turn a list of datetimes into a DimCoord.
    """
    unit = cf_units.Unit(unitstr, calendar)
    date_nums = unit.date2num(datetimes)
    return iris.coords.DimCoord(date_nums, units=unit, standard_name=standard_name)

datetimes = [datetime.datetime(2017, 6, day) for day in range(1,21)]
time1 = time_coord(datetimes)
time2 = time_coord(datetimes, unitstr='days since 2017-01-01')

Obviously the above is artificial, but this type of mismatch could easily occur with data from different sources.

An example from my recent work.  I have daily means of 6-hourly data from the UM, which come ready calculated with the time coordinate point at 12Z for each day.  For comparison, I download 6-hourly ERA-Interim data and use cube.aggregated_by to create daily means.  Having averaged over hours of 0, 6, 12 and 18, the ERA-Interim daily means have the time coordinate point at 09Z for each day.

OTOH, I did spend a lot of time back in October trying to figure out why my code was producing nonsense in its output files.  I eventually realised it was the exact issue illustrated by the new test_reversed_points() in #2498.  So I think I am in favour of a check on the coord points, but I would like a keyword that lets me bypass it if I'm sure my points are different for a good reason.

Note I like the idea of having a keyword which allows a user to bypass checks on points-matching.

Reviewing status for v2.0 inclusion
Not really Dask, but could be a behaviour change
so maybe for v2.0 it that will avoid another iris.FUTURE option

the behaviour was not consistent between the two groups of math ops (add/subtract had more strict coordinate matching checks than multiply/divide).

Just to stick my oar in ⛵ 😄 ...
Fundamentally the behaviour of multiple & divide should be different from add & subtract. Just consider the following:
speed * time = distance
speed + time = nothing meaningful

And for units:
meters * meters = meters ^ 2
meters + meters = meters

That means we need to do very different things to metadata in a cube depending on the operation.
We are in a bit of an unfortunate pickle with Iris on this regard. I think we are going to have to roll with Iris v2.0 in the same state as v1.13. I think we may want to fix the behaviour for v2.1 as we have broken an important use case.
I completely accept the code for this is far more complex that it needs to be (I take the blame for that), and we should put some effort in to rationalising it and sharing as many components between the operators as possible.

Just to be awkward, here is a use case for adding cubes with different units! 😉
More seriously, yes it's true that checks on the phenomena and units make sense for add/subtract but not for multipy/divide.  But I can't think of a reason why the coord checks should be different.

I think we may want to fix the behaviour for v2.1 as we have broken an important use case.

I'm pretty sure I'm not the only one who would rejoice if/when this is done.  Workarounds to force the checks to pass are not difficult to write, just fiddly to figure out which piece of metadata is tripping the check and needs deleting or copying to the other cube.

The heat index example is interesting, but of course it isn't really adding percentages to temperatures.
The definition of heat index it links to at http://en.wikipedia.org/wiki/Heat_index#Formula makes it clear that there is a constant in front of each term:
HI = c1 + c2.T + c3.R + c4.T.R + c5.T^2 + c_6.R^2 + c_7.T^2.R + c_8.T.R^2 + c_9.T^2.R^2
He has a shortened version : HI = temp + rh + temp*rh + temp**2*rh + rh**2*temp
So I think he has a whole lot of constant factors which happen to be all numerically equal to 1.0, but they must have units !
In Iris the 'correct' way would be to use scalar cubes with appropriate conversion units.
Very crude example:
>>> temp = Cube([295.0], 'air_temperature', units='K')
>>> rh = Cube([45.0], 'relative_humidity', units='0.01')
>>> t_per_percent = Cube([1.0], long_name='c1', units='K / 0.01')
>>> temp + rh
 . . .
iris.exceptions.NotYetImplementedError: Cannot use 'add' with differing units (K & 0.01)
>>> temp + t_per_percent * rh
<iris 'Cube' of unknown / (K) (-- : 1)>
>>> (temp + t_per_percent * rh).data
array([ 340.])
>>>

Sorry, is all this really obvious to everyone ?

@pelson

Fundamentally the behaviour of multiple & divide should be different from add & subtract.

This is true with regards to the metadata of the phenomenon (i.e. its units), and I believe we still have the behaviour you describe, but it doesn't apply to coordinate metadata, where I think the same logic should apply in all cases.

I just realised that #1987 is also related.

Awesome, thanks @ehogan, just what I was looking for 😺
@rcomer do you have any further use case examples that you want to contribute that hasn't already been captured here?

Another thing I don't think this issue covers, but was raised by @ehogan at #2698 is what happens to scalar coordinates that don't match?  Currently they just get chucked away.  Prior to #2498 we kept the coord from cube1 for multiplication and division.
I guess scalar coordinates don't necessarily need to be treated differently from other auxcoords.  But as they currently are, perhaps worth bearing in mind.

Currently [unmatched scalar coords] just get chucked away.

This was helpful for my debugging today, as the presence of a particular scalar coord told me that the subtraction operation that should have happened in fact had not.  🤔


how to move unused css rules to new file in grunt uncss task

Grunt uncss remove the unused css rules. My question is that is there a way by which instead of removing unused css rules, it will put them in a separate file

Nope, sorry. You should ask upstream for this feature.


Compiling on RPI4 using ws2811_sim.go

As stated, when compiling on a RPI4 it is using ws2811_sim.go.
I attempted to fix this by forking and changing the build constraints to be:

ws2811_arm.go to // +build arm arm64
ws2811_sim.go to // +build !arm,!arm64

After trying quite a few combinations of build constraints it is still falling into using sim.
My understanding of build constraints is limited, so I'm sure there is something simple I am overlooking.
I have verified that it is a build constraint, and not my failure to install the C library by combining all of the necessary code into ws2811.go and my code controls my neopixel array as expected.
go env:
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/home/sonnen/.cache/go-build"
GOENV="/home/sonnen/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/sonnen/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go-1.14"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go-1.14/pkg/tool/linux_arm64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/sonnen/projects/lights/lights_go/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build382712525=/tmp/go-build -gno-record-gcc-switches"

Thank you for your feedback. I haven't implemented the code for the ARM64 architecture yet because, until now, nobody seemed to need it. But with your issue, I will look into the matter.

Hey no problem. Thank you for the quick response. If there is any way I can help troubleshoot, or documentation you could point me to I could try to implement it and submit a PR.
I'm not sure why changing the build flags wasn't working. Is it something to do with the file name as well?

I don't think that the file names are relevant and what you did seems OK to me. I have to make some experiments. Are you compiling on the RPi 4 directly or are you cross-compiling ? And what OS/version do you have on the RPi 4 (Ubuntu, Pi OS) ?

Actually the file name might be relevant :
https://golang.org/cmd/go/#hdr-Build_constraints
----- BEGIN CITATION -----
If a file's name, after stripping the extension and a possible _test suffix, matches any of the following patterns:
*_GOOS
*_GOARCH
*_GOOS_GOARCH

(example: source_windows_amd64.go) where GOOS and GOARCH represent any known operating system and architecture values respectively, then the file is considered to have an implicit build constraint requiring those terms (in addition to any explicit constraints in the file).
----- END CITATION -----

So you can try to rename ws2811_arm.go to ws2811_hw.go (for Hardware)

Looks like the file name was the missing piece. I caught that in the docs after struggling all night trying to get it to work.

Thank you @onnenon. I just merged your PR in the master branch. Please, let me know if I need to do something else. If you are happy with the solution, could you please leave a comment and close this issue ? Thank you in advance.

Hey thank you. PR #14 resolves this. I'll go ahead and close it.

Hi @supcik! Just learning go and spent more time then I should have figuring out why I could not get a hello world app to run natively on a Raspberry pi 4. Turns out go mod is pulling v1.0.6 and I was reading master :)
Could you publish v1.0.7 when you have a chance?

Thank you @finack for your message. I just published the tag v1.0.7. Could you please let me know if you can now use the module without @master ?

Hi @supcik! got get pulled down the latest, v1.0.8! Thanks for the update!

👍🏼


Add FK from visualizations to users

cc @rafatower @Kartones

There has been no activity on this issue for more several months. We are closing it. If you think this still needs to be addressed please open a new issue.


Labevents without values (minor issue)

The following are minutiae, so please don't spend much time looking into them:
I noticed some inconsistencies regarding lab events, specifically regarding various lab events that have no data in the "value" or "valuenum" fields but which nevertheless have values in the comments field.
For example,  with lab itemID 51652, "HIV 1 viral load" in blood, there are many inconsistencies related to labevents.  For example, many "value" and "valuenum" fields are null.  Many of these values have comments which have the value of "DETECTED".  Most of these "DETECTED" rows (with no values) seem to have "flag" field values of "abnormal", but some have neither "value", "valuenum", nor "flag" values.
My questions are:

On rows with comments that read "DETECTED" but without values / flags (eg labevent_id 40242163), how were these presented in the source EHR lab screen?  If they were identified as abnormal in the original EHR, is there a missing data column that would identify them as such?  If the absence of "abnormal" flag indicates normal, is there a reason why some rows with "DETECTED <1.3" [log] values (ie about 20 copies) have abnormal flags and some don't?
Why are the comments for these lab values partially redacted?  For example, comments frequently read "DETECTED  <1.3*.  PERFORMED USING THE ___ HIV-1 TEST V2.0.  FOR RESULTS PRIOR TO ___ SEE MICROBIOLOGY TAB."  It seems the specific test is redacted, but the version number is not.  But given the "version 2" part, I would guess these tests are "COBAS AmpliPrep/COBAS TaqMan".  If the goal is to redact the actual assay brand/type, shouldn't the version number also be redacted?  (Although I'm guessing you are going to say that the de-identifier interpreted the assay name as possible PPI and redacted it for that reason.)

Interestingly, it seems that (for at least some lab itemid values) that Mimic IV has the actual values in the 'comments' column whereas Mimic III had them in the 'value' column.  For example, you can look at labevents for ITEMIDs 50827 and 50828.
I can't imagine that ALL of these instances are being represented in the front end by a footnote that requires clicking on to view the value.  I'm wondering if the EMR would, in some cases where the 'value' column was null, populate the result box with contents from the 'comments' column instead.  I'm not sure why they would do this, but if true, it would mean that some of the labevent values are actually in the comments column.

Hm, you're right, in the front end the values for 50827 (Ventilation Rate) appear in the normal location; not as a comment. The raw source of the data changed between MIMIC-III and MIMIC-IV; originally it was from the lab and now it is from a central data lake. I looked through the code for the MIMIC-III labevents and we didn't do any parsing of comments to extract values, so the data must have already been in the value column. I guess the central data lake doesn't have the same movement of data.
I think it makes sense for us to move some of these values from comments to value if we can think of a few simple rules to do so.

In the process of trying to map lab itemids to LOINC codes, I seem to run into more and more instances of this phenomenon.
In regards to rules for transferring the comments column to the value column, I would look at how the front end treats the various labevent rows where where the value is NULL but comments is NOT null, just to see if any of these still use a footnote type of notation to refer to the comments.  If none, maybe it's ok to move the comments to the value column.  (It's very hard to judge the content of the comments column on my end because it seems a whole LOT of the comments have been replaced with underscores by deid).
If (in the front end) there are at least SOME lab values (that meet previously mentioned criteria) that refer user to look at comments (ie footnotes), there is probably a hidden datum somewhere that tells the front end how to treat these values.   I would imagine this behavior (either footnote or replace-value-with-comments) would be uniform for a given ITEMID.  If not, maybe the front end is doing this based on the length of the comments for that row.


Bug when Freq_all is not specified

Traceback (most recent call last):
File "/bask/projects/v/vjgo8416-ms-img-pc/marji/affinity-vae/run.py", line 703, in run
train(
File "/bask/projects/v/vjgo8416-ms-img-pc/marji/affinity-vae/avae/train.py", line 536, in train
if config.VIS_CON and (epoch + 1) % config.FREQ_CON == 0:
TypeError: unsupported operand type(s) for %: 'int' and 'NoneType'

I receive this error when the Freq_all flag is not specified. Im not sure when or where this was introduced

I receive this error when the Freq_all flag is not specified. Im not sure when or where this was introduced

I think this is because in the config Freq_all is defined by default as None, maybe we should change to False. However, I'm not sure why the error is complaining on FREQ_CON, that has a default definition, but maybe is not set up correctly?

Somehow, freq_con gets set up as freq_all == None.

Im not sure why it wouldn't be?  There is a default value of 10 in config.py for all of the frequencies. and my log file says that it is setup:

Found the bug :)
https://github.com/alan-turing-institute/affinity-vae/blob/8c673018d5e7551c72cfefb655ac721fd695a658/run.py#L680

How did that get in there ??? :)). Thanks a lot. I fix it in my PR


dts: arm: st: f4: fix die_temp channel

The STM32F4 socs have different channels for the temperature sensor. Some are at channel 16 and some at channel 18. Made changes wherever it was relevant.
In short, the base configuration is to channel 16 and wherever it is supposed to be 18 it is overridden.

From what I gathered from the various F4 Ref Man (8 of them):

F405/407/415/417 use channel 16 exclusively
F412/413/423/427/429/437/439/446/469/479 use channel 18 exclusively
F401/410 can use either one
Not clear if F411 can use both or only 18 (channel 16 is sometimes mentioned, sometimes not)

So I'd say we can keep channel 18 by default, but set channel 16 in stm32f405.dtsi, and reset it to 18 in stm32f427.dtsi (which is already done). All other redefinitions could be removed.
What do you think?

I went according to modules/hal/stm32/stm32cube/stm32f4xx/drivers/include/stm32f4xx_ll_adc.h (lines 642-647).
According to that:

405/415/407/417/401/410 - Use channel 16
411/412/413/423/427/437/429/439/446/469/479 - Use channel 18
I then looked at the "inclusion tree" starting at stm32f4.dtsi. The only ones who include it are stm32f401.dtsi and stm32f410.dtsi which is why I thought the base stm32f4.dtsi should be configured to channel 16. I then kept going through the "inclusion tree" and changed whatver was needed which happened to be only stm32f412.dtsi!
When I saw there was only one file that needed to be updated I was kind of surprised that it was so easy so I looked at the commits and saw that it really used to be configured to channel 16 in stm32f4.dtsi until https://github.com/zephyrproject-rtos/zephyr/commit/ac7f2dad4ecba95ace2bc8473ff630e7707bf171.

In conclusion, unless I misunderstood something I think this is the correct way to have it.


List tests from swift-test

Updated this
depends on #406 #405 #403

LGTM.
Feel free to PTAM once the temporary file lands and this is rebased.

@swift-ci please test and merge

this just broke all my unit tests in four libraries i maintain… rip


hiprtc_paths is never closed

hiprtc_paths in hipew is loaded with dynamic_library_open_find but is never released when the program exit.
This might create a problem where some of the dlls are occupied by the processes that are in "exit state".

just fixed in the main branch of GPUOpen-LibrariesAndSDKs
https://github.com/GPUOpen-LibrariesAndSDKs/Orochi/commit/d40b1194ae6b7c3a667f3a20e8a25e9e7b2669f5


add quick guide for controlling a soft synth with orca

It would be great if people who found orca can very quickly start making sounds with it.
I saw there's some work getting started at #20 + Pilot. In addition to our toy synth, what do you think about having some helpful links for linking Orca with common soft synths? For me it took a while to search the net and figure out how to set up Orca => Midi => SuperCollider.
Links that were helpful to me: A guide about enabling IAC Driver, and SuperCollider's MIDIIn example.

Have you had a look at Tutorial.md? You could append what you think is missing to that page :)

wow great, that's what I'm looking for 👍


Enable snippets completions if it enabled in config

You already have config parameter for that, but it does nothing. This PR fix it.

Args formatting need to be improved - there are some problems with cases like function args.

@s-kostyaev
thanks


create /opt/lsst/daq-sdk/ccs-production symlink

Alternative to https://github.com/lsst-it/puppet-ccs_daq/pull/3


Refresh mapping reuse conn

Use existing connection for refreshing slots maps
This should speed up refresh_mappings. Existing connections are found in the pools. If no pools exist or if none are usable, a connection is established to one of the init nodes, as before.
This PR also gets rid of eredis_cluster_pool_worker. The eredis clients are used directly as a poolboy workers.
Before: The poolboy workers are eredis_cluster_pool_worker processes each which have an eredis process which have a socket.
After: The poolboy workers are eredis processes.

Thx for the comments.

For a future improvement: If a query from an application to a node/pool fails, we will refresh the mapping using the pool. Can we avoid using the failing node when attempting to get the mappings? At least try it last..this might avoid a timeout if the node/pool is really dead

Passing around the node which failed would add a bit of complexity, but sure, let's keep it in mind as a future improvement. (The pool name would need to be passed to eredis_cluster:handle_transaction_result/2 and then to eredis_cluster_monitor:refresh_mapping/1...)
In the scenario where one Redis node went down, the eredis instances for that pool will be in the reconnect loop and eredis:q/2 returns {error, no_connection} instantly in this case and then the next pool (next Redis node) is tried, so this scenario is fine.
On the other hand, if there is a network problem and the eredis client we get from the pool believes it's still connected (it didn't receive tcp_closed) and we get a timeout when trying eredis:q/2, it's a different story. Then, it would perhaps be beneficial to keep track of which pool has failed.


how to do so_reuseport implementation?

how to do so_reuseport implementation? to have multiple salvo listening to same port

like this? https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Rust/salvo/src/main_pg.rs


enhancement - explore adding quicklook for vscode for osx

VSCode Version: n/a
OS Version: n/a

Steps to Reproduce:

click space on python file

it doesn't show with syntax highlighting

https://github.com/sindresorhus/quick-look-plugins

This seems to be doable—I see TextMate 2 handle previews for all source code.

https://github.com/joaomoreno/gulp-atom-electron/pull/46 allows this to happen. Setting it for November.

Awesome, thanks for the fast pull!

I'm sure this will be great for tonnes of users.
I have this older ticket - but it got shelved.
A variable explorer like spyder.
https://github.com/Microsoft/vscode/issues/29971
@agermanidis made a fantastic effort in surfacing these here
https://github.com/agermanidis/livepython
I wonder how hard it would be to rip apart the debug variable explorer.
I don't know where to start.  A friend illustrated to me 6 mths back that the content inside vscode could be inspected like google chrome with divs and dom hierarchy.

@rsesek The changes don't seem to have made any change. For example, after installing today's Insiders, .ts files still don't show source in QuickLook, although they are associated with Code.

LaunchServices may not have picked up on the change, and it's a bit fiddly (especially if you have multiple apps with the same CFBundleIdentifier on the machine). It also won't work if the CodeSignature is invalid.
/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -f /path/to/Code.app
I downloaded Insiders today (1.18.0-insider) and didn't see the new key in the Info.plist, FWIW.

I'll take a look at it in November. 👍

@rsesek Are you sure all this needs is to add the LSItemContentTypes key? I can't get this to work on my machine in any way.
Also, the official docs say:
LSItemContentTypes contains an array of strings with the UTI types that represent the supported file types in this group.

The LSItemContentTypes key identifies the UTI associated with the file format and the CFBundleTypeIconFiles key points to the icon resources to use when displaying it.

I get the feeling that this is not sufficient for Quick Look, but rather that a Quick Look plugin is required to get this to work.

OK guys, I'm going to close this and revert the PR for two reasons:

Causes issues https://github.com/Microsoft/vscode/issues/39496
There are other alternatives out there which do a much better job than Code: https://github.com/sindresorhus/quick-look-plugins


Anaysis on meals based on presence information

This notebook sums up all of the knowledge I gathered over time around identifying suspicious meals:

Based on information about presence in Brasilia or abroad;
Taking into consideration the use of the "undocumented OCR data" from https://github.com/datasciencebr/serenata-de-amor/pull/207 to grab receipts timestamps;
And also introducing a rudimentary scoring mechanism that could be expanded in case this type of knowledge is implemented into Rosie.

It includes "unreleased" datasets from both https://github.com/datasciencebr/serenata-toolbox/pull/20#issuecomment-304748817 and an updated version of the code in my spike on a scraper for official missions which I plan to submit as a PR to the toolbox not long after my presences refactoring PR is merged.
The new official missions dataset scraping logs and code can be found here (used an old version of my toolbox fork) and the CSV for it is 2017-04-19-official-missions.xz.zip
As much as I hate to propose yet another PR that might take a while to be merged, I feel like this should be on you guys' radar as well 😄
/cc @rodolfo-viana @Irio @cuducos

Did you have the chance to check for the percentage false positives? This would be a great way to close the notebook.

@Irio I didn't when I put this together because the "receipts PDFs server" was giving me back 500s.
After a few clicks today it seems that it does gives us back some false positives as it bumps up the scores of receipts like the following:

http://www.camara.gov.br/cota-parlamentar/documentos/publ/3118/2016/6136366.pdf
http://www.camara.gov.br/cota-parlamentar/documentos/publ/1586/2016/6099888.pdf

Those PDFs include some sort of report to say that the "nota fiscal" is valid which might be emitted in a later date and the resulting PDF include a hell lot of timestamps that confuses the logic I put together here.
Any thoughts on how to improve this? We do have a dataset for each reimbursement with individual page text but it is not in an usable format (can be derived from the raw google API OCR responses JSON). We could maybe lower the score if the receipt has multiple pages or if it has a huge amount of timestamp matches 🤔  There are really good suspicious reimbursements at the top of those results so if this needs some fine tuning it is going to be really well thought. To be honest, I think this kind of thing might only yield like really accurate results with some advanced computer vision in place.
In the meantime, I'll make the tweaks you mentioned and will add a conclusion with the "accuracy rate" & limitations of this approach.

@Irio

Did you have the chance to check for the percentage false positives? This would be a great way to close the notebook.

Finished that now, there are about 47 bad reimbursements out of about 60 reimbursements included in the analysis. As expected, the ones that are most suspicious are the ones that we have lots of timestamps found with OCR and in days that deputies spent a lot of time in DF. Also, now that the official missions dataset includes information about canceled trips, there's a much smaller set of bad reimbursements in that front and many false positives.
I'll add all my findings to the notebook (hopefully) before EOD today

Thanks, @fgrehm. This has everything to soon become a Rosie classifier.


Block explorer data

Block explorer data deserves first class support in The Graph. Not every node needs to index block explorer data but those that want to should be able to efficiently. Block explorer data includes indexing:

Blocks
Transactions
Accounts
Receipts

This should be possible without having to define a subgraph specifically for this data. Rather this should be specified as a CLI argument when starting up the node.
Questions:

Should we automatically index internal transactions?
What options should we enable for sharding? Should we skip this for a first pass?
How do we namespace these entity types? Should there still be GraphQL types that live in a subgraph?

@tarrencev expressed the wish to call contracts from the graphql client, so that he could get the latest state of the contract without having to index it. This can be done Etherscan, seems reasonable to put it on the wishlist for our block explorer functionality.

Rationale
I think Yaniv's original description covers this. We need this for users to be able to query blockchain-intrinsic data from Ethereum (blocks, transactions, accounts, transaction receipts). We also want subgraphs to be able to reference this kind of information from their subgraphs.
Requirements

Supports at least:

blocks
transactions
transaction receipts
logs
accounts (with balances).

Comes in the form of a built-in subgraph, served at /subgraphs/ethereum and /subgraphs/ethereum/graphql (GraphiQL), similar to the /subgraphs subgraph. Supports the same GraphQL API as we use for all subgraphs.
Can be enabled with a --ethereum-subgraph command line flag for Graph Node. Disabled by default.
All indexed Ethereum data can be referenced from other subgraphs.

Proposed User Experience
Querying block explorer data
After started with --ethereum-subgraph or ETHEREUM_SUBGRAPH=true, the Graph Node indexes the entire chain from the genesis block to the latest block, and then follows the chain as new blocks are being added.
Users can access the data by going to http://localhost:8000/subgraphs/ethereum/graphql (GraphiQL) or by using http://localhost:8000/subgraphs/ethereum (or the WS alternative) in their apps. They can send queries like the following to this endpoint to query Ethereum data:
{
  blocks(where: { number_gte: 0, number_lt: 1000 }, orderBy: number) {
    hash
    transactions(orderBy: gas, orderDirection: desc) {
      hash
      receipt { ... }
    }
  }
  accounts(where: { address: "0x..." }) {
    balance
  }
}

Referencing block explorer data
From a user's perspective, whether data comes from one subgraph or another should not matter. Assuming a field owner: Account! refers to the Account entity from the built-in Ethereum subgraph, a query like
{
  domains {
    owner {
      balance
    }
  }
}

should just work™. This includes being able to introspect the Account type and any related types in the GraphQL playground.
From a subgraph developer's perspective, the main novelty is subgraph composition. Given a subgraph name or deployment ID, types from the subgraph with that name or deployment ID can be imported and referenced in the GraphQL schema using a new @import directive:
@import(
  type: 'Account',
  as:   'EthereumAccount', # Optional renaming
  from: { subgraphName: 'ethereum' }
)

type User @entity {
  account: EthereumAccount!
}

Open Questions

Should subgraph composition allow subgraph mappings to access instances of the imported entity types in the store?
An alternative to the @import directive would be an import comment syntax a la https://github.com/prisma/graphql-import

Proposed Implementation
TODO
Open Questions

When composing subgraphs, how do we make introspection work? We'd have to merge schemas basically but that may result in type conflicts.

Proposed Documentation Updates
TODO
Proposed Tests / Acceptance Criteria
TODO
Tasks
TODO

Nice job putting this together!
My input on some of the open questions:

Should subgraph composition allow subgraph mappings to access instances of the imported entity types in the store?
My vote would be no. We should assume that in the long term relationships across subgraph entities will be plentiful, and requiring an Indexer to have all the subgraphs indexing (or available) in order to run the mappings for a single subgraph would be untenable. For the block explorer use case specifically, I believe we already expose much of the data in the mappings anyways.
An alternative to the @import directive would be an import comment syntax a la https://github.com/prisma/graphql-import
I like the directive syntax better than the comment syntax option. I presume that I could supply: from: { id: Qmsdf58... }  if I wanted to reference a subgraph by ID rather than by name?
Referencing and querying entities across the subgraph boundary could lead to querying both subgraphs at different times/blocks. What do we do here?
I think by default all subgraphs should be queried as of the same block. Currently we only support querying as of the "latest" block but we should also support querying as of a block supplied by the user in the query. Eventually, it should be possible to specify different blocks to query as of when traversing entity relationships. Not sure what the right stop gap is for right now, but I don't think it's desirable to have state queried across different blocks w/o this intention being expressed by the user.
When composing subgraphs, how do we make introspection work? We'd have to merge schemas basically but that may result in type conflicts.
Could we make the as field in the @import directive supply a namespace for the entire imported subgraph schema, rather than just alias a specific type?
Should we go straight to custom ID fields (e.g. hash and number for blocks) or stick to id for the initial pass? (My recommendation: stick to id initially.)
I'm also a fan of making id the block hash initially, and make number a normal attribute (which could return multiple blocks if we have uncles/ forked blocks in our DB).

We could make double underscore reserved in type names, so that we always have it available to prefix imported types with a namespace.
i.e.,
@import(
  from: { name: 'ethereum' }
  as: 'Eth', # Optional renaming
)

type User @entity {
  account: Eth__Account!
}

Other feedback/questions:

I'm a little bit hesitant to bake Ethereum any deeper into graph-node, with a special CLI flag, endpoint, etc. But I suppose we have a bigger refactor in store, anyways when we switch to the multi-blockchain architecture.
Do we have an API in mind for traversing reverse relationships at query time?

{
  domains {
    owner {
      __from( name: 'ens' ) {
        ownedDomains {
          names
        }
      }
    }
  }
}

What should the behavior be if a user runs graph-node locally and deploys a subgraph locally which references block explorer types? Should we automatically run the block explorer subgraph as well, or simply fail gracefully at query time? We will have to answer this question in the general case when we support subgraph composition as a first-class feature.

@Zerim Thanks for all the comments, I've incorporated them into the @import design and as decisions under the open questions.
About the other feedback/questions:

I think it'll just have to be that way for now.

I haven't thought about reverse relationships yet.

I'd handle that gracefully by just returning null for those references.

This here causes me headaches:

Should we go straight to custom ID fields (e.g. hash and number for blocks) or stick to id for the initial pass? (My recommendation: stick to id initially.)
I'm also a fan of making id the block hash initially, and make number a normal attribute (which could return multiple blocks if we have uncles/ forked blocks in our DB).

To not blow up storage, we can only store entities when they change; for time-travel queries, we need an efficient way to find the latest version of a given entity before some point in time. That's easy if we only store whatever we think the main chain is at any point in time, since blocks then have a total order. With uncles, that becomes a lot harder. Maybe we can do something clever with lexicographic ordering by (block number, sidechain number) where the main chain has always number 0, and each uncled block becomes its own sidechain. But there's a bunch of detail to be worked out, and we have to carefully look at the kinds of queries we need to support for uncled blocks and see if there are simpler ways to support time-travel in the presence of uncled blocks.

@lutter I think there's two separate questions here:

How to handle querying any subgraph as of a certain block (this is complicated by being able to query as of an uncle block, as you mention).
How to represent uncled blocks as entities in our block explorer data model.

I think we can follow my recommendation for #2, w/o it forcing a specific design on #1.

@Zerim one thing I don't understand about uncles is that they only need to have valid block headers, which means to me that that is all you can reliably query about uncled blocks. That to me means that they are not full blocks, and we should treat them as additional data attached to a block on the main chain. That wouldn't preclude us from supporting queries by block number that return information about uncles, but it does mean that uncles and blocks on the main chain are treated differently.

@lutter That's correct, unless we had seen an uncle block when it was published, or a forked block before it was reorged, we would only have header information. Which is why, for example, that's all you see on etherscan for forked blocks: https://etherscan.io/blocks_forked
There's a question as to whether if we have all the information for a block that is later forked/uncled, we should retroactively remove everything but the header to keep consistent with other uncled blocks we know of.
For very recent blocks, I'm sort of partial to keeping around as much information as possible, and then maybe pruning the remaining data when were confident that the block would be permanently forked/uncled.

I'm concerned about the proposed implementation strategy of basically indexing all of the data contained in an archive Ethereum node. That's currently 3 TBs of data when stored as compressed RLP in key-value storage. If we store this as heavily indexed relational data, that will be over 10 TBs. That is a serious operational cost.
I'd instead suggest that this subgraph is implemented by leveraging an archive node, instead duplicating the data from it, and exposes only the queries that we can efficiently resolve through the JSON-RPC interface.

@leoyvens I'll come up with an estimate of the storage this would occupy. A rough guess based on 10M blocks with 150 transactions each would involve maybe

10M block entities
300M transaction and transaction receipt entities
2M account entities (probably less)

That doesn't feel too excessive. Having this data available in the local database would mean

better indexing performance (don't need eth_getLogs, assuming all past blocks are already ingested)
block explorer data queries (don't have to go to Ethereum nodes)
efficient query-time composition (don't have to go to Ethereum nodes)

The GraphQL API built into geth will take a while to make it into Parity (if it ever will). It would help with query and query-composition performance, but we can't wait for it. Unless block explorer data requires TBs of data, IMHO the benefits we get from ingesting all this data outweighs the storage cost.

I agree that blocks and accounts are something we should just ingest and have great query performance for. However I'd like to make a point that transaction receipts may be a step too far.
Right now we have all transaction receipts loaded in our Graph nodes, storing a total of 330GB. We don't need to do this and I intend to get rid of virtually all of those by doing what is described in this comment,  'Proposed Implementation' section.
This opens the question of whether we should have transaction receipts as part of the subgraph discussed in this issue. First, I'd like to separate the concept of a full block explorer subgraph from a blessed 'Ethereum subgraph' for subgraphs to compose with.
For a full block explorer entities such as Log and Receipt are a requirement, and I believe we should eventually make it possible to build a block explorer subgraph that includes receipts, for the nodes brave enough to run those.
For the Ethereum subgraph to be widely composed with I agree it should be featureful and fast to query, but we also need to keep indexing costs down for there to be a good supply of index nodes and low query prices. I believe the entities Account, Block and Transaction will cover 99% of the use cases, while adding receipts and logs would cover the remaining 1% of use cases for 99% of the cost.
By not including receipts, we could have every index node sync this data by default, allowing us to assume and leverage the data in the internals of graph-node.

@leoyvens I agree with that, although I expect a few of the transaction receipts fields to be crucial enough so that we have to pull them in (thinking about the gas info for instance, which is split across the tx and the tx receipt).

@leoyvens There is one argument for storing logs though: almost every subgraph today defines entities that correspond to the event types and series the events almost 1:1.
If we can allow developers to just reference already existing event entities in the block explorer data, then that would save everyone a ton of time and work.
One aspect that slightly weakens this argument is that subgraphs typically only store a subset of events as entities, not all of them.

@Jannis Having the spent gas is fine, my concern is the logs.
The logs in a block explorer are not decoded, so having all logs is not the same thing as having a subgraph that ingests events with a proper schema.

Revised plan without subgraph composition.
Requirements

Supports at least:

blocks
transactions
transaction receipts
internal transactions
logs
contracts (with balances)
accounts (with balances).

Comes in the form of a built-in subgraph, served at /subgraphs/ethereum/mainnet and /subgraphs/ethereum/mainnet/graphql (GraphiQL), similar to the /subgraphs subgraph. Supports the same GraphQL API as we use for all subgraphs.
Can be enabled with a --network-subgraphs ethereum/mainnet ... command line flag for Graph Node. Disabled by default.
All indexed Ethereum data can be referenced from other subgraphs.
This importing approach drives subgraph index and replaces the block ingestor.

Proposed User Experience
Querying block explorer data
After started with --network-subgraphs ethereum/mainnet, the Graph Node indexes the entire chain from the genesis block to the latest block, and then follows the chain as new blocks are being added.
Users can access the data by going to http://localhost:8000/subgraphs/ethereum/mainnet/graphql (GraphiQL) or by using http://localhost:8000/subgraphs/ethereum/mainnet (or the WS alternative) in their apps. They can send queries like the following to this endpoint to query Ethereum data:
{
  blocks(where: { number_gte: 0, number_lt: 1000 }, orderBy: number) {
    hash
    transactions(orderBy: gas, orderDirection: desc) {
      hash
      receipt { ... }
    }
  }
  accounts(where: { address: "0x..." }) {
    balance
  }
}

Referencing block explorer data
From a user's perspective, whether data comes from one subgraph or another should not matter. Assuming a field owner: Account! refers to the Account entity from the built-in Ethereum subgraph, a query like
{
  domains {
    owner {
      balance
    }
  }
}

should just work™. This includes being able to introspect the Account type and any related types in the GraphQL playground. From a subgraph developer's perspective, the main novelty is subgraph composition. This work is tracked in #1202.
Proposed Implementation
Graph Node

Add a GraphQL schema for built-in ethereum network subgraphs.
Add an --network-subgraphs CLI flag to graph-node.
Add a NetworkIndexer component trait to graph.
Add an EthereumNetworkIndexer component to datasource/ethereum.

This component can be used as an alternative to the existing BlockIngestor, allowing our ingestion of Ethereum block explorer data to drive subgraph indexing in the same way.
The indexer detects reorgs and marks blocks that are no longer part of the chain as uncled.

For every network name passed to --network-subgraphs, anEthereumNetworkIndexer is created instead of a BlockIngestor.
Because the EthereumNetworkIndexer will be writing blocks constantly when catching up, change ChainHeadUpdateListener to poll for the latest block in repeated intervals as long as the latest block we have is far enough behind the head of the chain.

Open Questions

Should we go straight to custom ID fields (e.g. hash and number for blocks) or stick to id for the initial pass? (My recommendation: stick to id initially.)

Decision: id for the initial pass.

Proposed Documentation Updates

Document the --network-subgraphs CLI flag in the graph-node README.
Document the /subgraphs/ethereum/mainnet endpoints and the GraphQL schema on https://thegraph.com/docs.

Proposed Tests / Acceptance Criteria

Test the basic ingestion of all types of data using a mocked EthereumAdapter.
Test querying the Ethereum subgraph endpoint against a store with test data.

Tasks
The plan is to implement this in different phases:

Basic: The EthereumNetworkIndexer runs alongside BlockIngestor and only ingests the following data: blocks, transactions, transaction receipts and logs. The --network-subgraphs flag is supported and subgraphs can be queried via e.g. /subgraphs/ethereum/mainnet. The indexer handles reorgs by marking blocks as no longer being on the chain when they are reorg-ed out of the chain.

Accounts: The EthereumNetworkIndexer also indexes the following data: internal transactions, accounts, contracts, balances.

Ingestion: The EthereumNetworkIndexer acts as a drop-in replacement for BlockIngestor. This is activated for any network passed to --network-subgraphs.

The below estimates use fibonacci-style sizes:
1 ~ 0.5-1 days
2 ~     1 days
3 ~ 1.5-2 days
5 ~   2-3 days
8 ~    3+ days

Phase 1 (Basic)

[ ] Add a GraphQL schema for Ethereum network subgraphs (1)
[ ] Add --network-subgraphs CLI flag (1)
[ ] Add NetworkIndexer trait to graph (1)
[ ] Add basic EthereumNetworkIndexer without reorg support (2)
[ ] Enable EthereumNetworkIndexer for all requested networks (1)
[ ] Add reorg support to EthereumNetworkIndexer (2)
[ ] Add tests for ingesting all Ethereum data types using a mocked EthereumAdapter (2)
[ ] Add tests for reorg handling using a mocked EthereumAdapter (2)
[ ] Document the --network-subgraphs CLI flag in README (1)
[ ] Document the --network-subgraphs and network subgraph endpoints in the docs. (1)

Phase 2 (Accounts)

[ ] Index internal transactions in EthereumNetworkIndexer (5)
[ ] Index accounts and contracts in EthereumNetworkIndexer (without balances) (3)
[ ] Update balances of accounts and contracts based on transactions and internal transactions (5)
[ ] Add tests for ingesting internal transactions using a mocked EthereumAdapter (2)
[ ] Add tests for ingesting accounts and contracts using a mocked EthereumAdapter (2)
[ ] Add tests for updating account/contract balances (including internal transactions) (3)
[ ] Document how to query internal transactions, accounts, contracts, balances (1)

Phase 3 (Ingestion)
tbd

@Jannis from that plan, it sounds like we're back to having raw logs in the standard Ethereum subgraph, is that the case?

@Jannis Has this project been tabled/abandoned. Right now I'm working on a project to retrieve all uniswap transaction history for taxes and having the ability to automatically query gas fees paid on each transaction would be incredibly valuable.
If so, is there a workaround or alternate path forward to retrieve the gas used on each transaction without having to also make separate calls to Etherscan or another API?
Thanks!

@lucaswalter It's only alive in telegrams and price action, the official discord, team doesn't response, just a bunch of hooligans joking around. The most fundamental of use case such as yours, and very similarly mine is abandoned .

@Jannis Has this project been tabled/abandoned.

It has been paused but not abandoned permanently. Adding transaction indexing to the existing codebase wouldn't actually be that hard. The difficult part (more difficult than it looks in the plan) is having accurate account balances, because it likely will require replaying block rewards and internal transactions, which will make things extremely slow.

Thanks for the quick reply @Jannis. For our use case (and really all accounting applications), having access to the fees paid across all subgraphs is necessary.
Is there a path forward to retrieve gas fees for transactions on a given subgraph within the graph ecosystem right now?

@Jannis Are you able to share what this has been paused in favor of? This seems fundamental to the success of TheGraph. Transactions are where the action happens on chain. TheGraph is being compared as the "Google of Blockchains". Keeping with the analogy, returning only blocks is akin to Google only returning domains instead of actual webpages.

@Cooksauce Stratis, the 3rd highest hyped project and gainer in 2017, was the "Microsoft of Blockchains", check it out today. They have had to rebrand to re-hype. It had very similar vibe to this one. But the discord community wasn't as low quality. I was looking upto this project from real usability perspective but it has been a sheer disappointment.

@Jannis Wanting to follow up here if there is an ETA or if there is any workaround for retrieving transaction gas fees via The Graph. I did come across this sub-graph which claims to return fees but the transactions don't return anything when querying: https://thegraph.com/explorer/subgraph/sistemico/eth-gas
Thanks!

Very disappointing we can't even get a response to a simple question here. Does not instill a lot of trust in the graph ecosystem. We will be pursuing alternatives.

Will this allow querying transactions for a specific address, with a given data parameter?
Something like this:
{
  transactions(where: { to: "0x...", data: "0x1234abcd" }){
     value
     from
     hash
  }
}

am working on a project that would greatly benefit from access to internal txns -- is this still on the roadmap somewhere or has it been abandoned now?

When is this going to be live?


pipeline panic

This has panicked a couple times in the last hour or so running in prod
 panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1ac3437]

goroutine 2571 [running]:
github.com/buraksezer/olric.(*DMapPipeline).execOnPartition(0xc000621800, {0x29e4ba8, 0xc001650540}, 0xc000c998e0?)
	/Users/derek/go/pkg/mod/github.com/buraksezer/olric@v0.5.1/pipeline.go:483 +0x317
github.com/buraksezer/olric.(*DMapPipeline).Exec.func1()
	/Users/derek/go/pkg/mod/github.com/buraksezer/olric@v0.5.1/pipeline.go:532 +0x6e
golang.org/x/sync/errgroup.(*Group).Go.func1()
	/Users/derek/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64

Hello @derekperkins,
I couldn't manage to reproduce the problem. It's very likely there is a nil item in the list of commands. pipe cannot be nil. I reviewed the code. I couldn't find any reason for nil commands. There can be a subtle bug in the command pool logic. Do you use Discard?
func (dp *DMapPipeline) execOnPartition(ctx context.Context, partID uint64) error {
	rc, err := dp.dm.clusterClient.clientByPartID(partID)
	if err != nil {
		return err
	}
	// There is no need to protect dp.commands map and its content.
	// It's already filled before running Exec, and it's now a read-only
	// data structure
	commands := dp.commands[partID]
	pipe := rc.Pipeline()

	for _, cmd := range commands {
		pipe.Do(ctx, cmd.Args()...) --> panics
	}

Could you share a code snippet to reproduce the problem?

Here is my hypothesis:
execOnPartition assumes that dp.commands is read-only during the execution. Neither dp.commands nor its content is protected by a mutex during the execution. If your code calls Exec and Discard simultaneously, execOnPartition may encounter nil items in dp.commands.
Discard calls dp.closedCancel() at line 548, if Exec is already passed the context check at line 503, they can start running parallel.

Yes, we're using Discard and reusing pipelines, but we're not calling Exec and Discard simultaneously on our own accord. This is simplified, but basically this is what we're doing.
defer pipeline.Discard()

pipeline.Exec(ctx)

for _, futureGet := range futureGets {
    futureGet.Result().Scan()
}

I'm wondering if there's a race condition when context is canceled...

I'm wondering if there's a race condition when context is canceled...

I believe the possible root cause is this.


[Feature Request] Change to protected virtual method ClaimsIdentity CreateclaimsIdentity

Change to protected virtual method ClaimsIdentity CreateclaimsIdentity(JsonWebToken, TokenValidationVarameters, Issuer) to allow SAL to override method so that it can do claims mapping.
Why?
Microsoft.IdentityModel has two assemblies to manipulate JWT tokens:
System.IdentityModel.Tokens.Jwt, which is the legacy assembly. It defines JwtSecurityTokenHandler class to manipulate JWT tokens.
Microsoft.IdentityModel.JsonWebTokens, which defines the JsonWebToken class and JsonWebTokenHandler, more modern, and more efficient.
When using JwtSecurityTokenHandler, the short named claims (oid, tid), used to be transformed into the long named claims (with a namespace). With JsonWebTokenHandler this is no longer the case, but when you migrate your application from using JwtSecurityTokenHandler to JsonWebTokenHandler (or use a framework that does), you will only get original claims sent by the IdP. This is more efficient, and occupies less space, but might trigger a lot of changes in your application. In order to make it easier for people to migrate without changing their app too much, this PR offers extensibility to re-add the claims mapping.

Closing as issue was already created in #1894


Exception error

I'm getting error given below:
Could not load file or assembly 'xxxxx, Version=1.0 Culture=neutral, PublicKeyToken=null' or one of its dependencies. Attempt to load an unverifiable executable with fixups (IAT with more than 2 sections or a TLS section.) (Exception from HRESULT: 0x80131019)

Checklist:

[X] I have checked that I am dealing with an EazFuscator binary, and believe that it can't be another obfuscator such as ConfuserEx.
[-] I understand that an error message may mean that a certain protection was not present.
[X] I can share a sample that can reproduce the bug

Your executable is not a valid .NET binary (likely if you dumped it incorrectly), or there is a bug in dnlib meaning it cannot load your executable correctly. In either case, this is not an EazFixer bug.

But DiE says it is. Can it be copycat?

It looks like the executable has some modifications made to it that differ from standard EazFuscator, making it so it can't be loaded with reflection. This isn't something you can fix without patching the executable itself. You can try running it through de4dot but I cannot guarantee that EazFixer will ever work on this executable.

de4dot also couldn't decompile methods, procedures etc.
What's more, when I try to debug with ollydbg the executable closes itself.
Do you have any extra tricks to decompile and "patch" my silly software?

No. There is more to this executable than just standard EazFuscator so it's out of scope for this project, and this issue thread is not a support forum. I suggest going to a forum dedicated to reverse engineering and asking for help there.


Add new filter by campaign; re-factor code to match common name patterns for sorting/filter functions; add changes to templates

fixes #7704

Thanks Valentin, I will run the test.

Valentin, There is a couple of errors.
https://reqmgr2-dev.cern.ch/reqmgr2/approve
Campaign is not showing (next to Group).
Data is not populated when page is initially updated. Only after click the sort option.

Seangchan,
could you please re-apply the PR, I added few more changes which I missed
originally. This time campaign will show up along with default page loading.
And, I need you to inject data with different campaigns such that we'll test
the filter.
Thanks,
Valentin.
On  0, ticoann<EMAIL_ADDRESS>wrote:

Valentin, There is a couple of errors.
https://reqmgr2-dev.cern.ch/reqmgr2/approve
Campaign is not showing (next to Group).
Data is not populated when page is initially updated. Only after click the sort option.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/7719#issuecomment-286509474

Thanks Valentin, But search still doesn't work. I added new campaign name.
https://reqmgr2-dev.cern.ch/reqmgr2/approve

Seangchan,
could you please apply PR again and re-deploy, it seems like I forgot to change
the filter name and campaign search didn't work.
Let me know, when it's ready.
Thanks,
Valentin.
On  0, ticoann<EMAIL_ADDRESS>wrote:

Thanks Valentin, But search still doesn't work. I added new campaign name.
https://reqmgr2-dev.cern.ch/reqmgr2/approve
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/7719#issuecomment-286538174

Thank Valentin, Now it works fine. Could you squash the commit?

test this please

Instead of Filter by pattern..., Could you change to Filter by request name?

Seangchan,
what's the point? The pattern is more flexible. I don't mind to change it but I
think pattern is more interesting from end-user point of view.
Best,
Valentin.
On  0, ticoann<EMAIL_ADDRESS>wrote:

Instead of Filter by pattern..., Could you change to Filter by request name?
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/7719#issuecomment-286629451

Valentin, pattern is against request name field right? no other properties, correct?
I think it is more clear to specify which fields they are searching it for. so first box is search against request name and second box is search against campaign.

Valentin, pattern is against request name field right? no other properties, correct?
yes

I think it is more clear to specify which fields they are searching it for. so first box is search against request name and second box is search against campaign.

ahh, I think you want to change text in red within search field, e.g.
replace
Filter by pattern ...
change to
Filter by request name
while Filter by campaign is a right text, Filter by campaign, and it does
search Campaign field of request but it does that by pattern.
So, I changed PR, please take it and redeploy and let me know if you're
satisfied.

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/7719#issuecomment-286763235

ahh, I think you want to change text in red within search field, e.g.

Sorry I wasn't clear. Yes I just want to change in the display in the text box.

It looks good to me Valentin, Once it gets in testbed next well. I will ask @fabozzi to check.
When you squash the commit, I will merge that. Thank you very much.

FYI @fabozzi, this is the added search campaign field

Done, squashed.
On  0, ticoann<EMAIL_ADDRESS>wrote:

It looks good to me Valentin, Once it gets in testbed next well. I will ask @fabozzi to check.
When you squash the commit, I will merge that. Thank you very much.
--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/dmwm/WMCore/pull/7719#issuecomment-286796370

Unittest failure doesn't seem to be related. Will investigate separately.


Enhanced APC UI + New NT logo to footer

About the PR

This updates the APC UI to use the new FancyWindow and other minor tweaks to make the UI just a bit nicer.
This also adds a new generic NanoTransen logo that can be used in footers or anywhere else people may need it.
Media

Old UI:

New UI in this PR:

If no access

[x] I have added screenshots/videos to this PR showcasing its changes in-game

Changelog

:cl:

tweak: APC UI gets a visual update
tweak: APC will now disable its toggle and show a tooltip if you don't have access.

Any outstanding issues @AJCM-git ? I think this is ready to merge.

I recently just got myself confused after forgetting about this; "Battery Power" is really unsuitable for this and "Load" is better. I will PR a change.