ADC Voltage Reference aren't used correctly

I saw a strange behavior while using ADC. If I first set ReferenceVoltageSource.Internal and than GPIO3Function.ADC3 the settings seams to be ignored. This test was done with a fix voltage on GPIO3 (ABOUT 3.9V so 938 seams right at 4096mV ref)
(Pdb) self._mcp2221.chip.read_adc(2)
815
(Pdb) self._mcp2221.chip.write_adc_reference_source(ReferenceVoltageSource.Internal)
96 0 0 0 0 135 0 0 16 8 8 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
(Pdb) self._mcp2221.chip.read_adc(2)
938
(Pdb) self._mcp2221.chip._write_sram(SramDataSubcode.GPSettings, 3, GPIO3Function.ADC3)
96 0 0 0 0 0 0 128 16 8 8 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
(Pdb) self._mcp2221.chip.read_adc(2)
815

All readbacks of SRAM via 0x61 where fine. Also  'read_adc_reference_source`.Maybe a bug in the CPU?
I saw a workaround for gpio, not sure if something like that it is needed for "ADC Voltage Reference" too?

Ok yes the setting change goes through _sram_write as well. Your fix should have fixed that reference change too.

Actually this is in the SramDataSubcode.ChipSettings section and not covered by my change. From datasheet I would say the reference should not be touched because of
Bit 7: Enable loading of a
new ADC reference
1 Bits 2-0 will be used for ADC reference voltage selection
0 ADC reference will remain unaltered

in "Set SRAM settings"
It feels a bit wrong to set this bit and force a rewrite of the configuration.

Yes I see, and it wouldn't help since you say the voltage source setting remains in SRAM the way it should. What puzzles me is that both of your readings are not that close to 3.9V (should be 975 with a 4.096V ref). Feels more like an impedance issue to me. I'd play with the source impedance maybe.

Thanks for your suggestion. The values are quite repeatable.
My VDD is about 4675mV and the meassured voltage about 3815mV. It comes over a resistor divider 10k + 1k875 from a 24V source.
So I should get
3815mV / 4096mV * 1024 = 954
3815mV / 4675mV * 1024 = 835
What I see is
(Pdb) from mcp2221.enums import ReferenceVoltageSource, GPIO3Function
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.write_adc_reference_source(ReferenceVoltageSource.Internal)
(Pdb) self._mcp2221.chip.read_adc(2)
945
(Pdb) self._mcp2221.chip.read_adc(2)
940
(Pdb) self._mcp2221.chip.read_adc(2)
940
(Pdb) self._mcp2221.chip.read_adc(2)
939
(Pdb) self._mcp2221.chip.read_adc(2)
942
(Pdb) self._mcp2221.chip.read_adc(2)
941
(Pdb) self._mcp2221.chip.read_adc(2)
944
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.gpio3_write_function(GPIO3Function.ADC3)
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
819
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.gpio3_write_function(GPIO3Function.ADC3)
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
819
(Pdb) self._mcp2221.chip.read_adc(2)
820
(Pdb) self._mcp2221.chip.read_adc(2)
820

820÷1024 × 4675mV = 3743 mV
942÷1024 × 4096mV = 3768 mV
So they both are equally lower. This may be an impedance issue. We are quite over the max 10k and have no capacitor.
But Setting the gpio Function to ADC3 as it was set before will change the result in the same way as switching the Reference to VDD. Setting the Reference actively to Internal heals this.
I guess for my usecase I can set the reference_source again as poor workaround.

I tried with my device, which has pin 1 connected to 3.3V through a 10k resistor and pin 3 through a 100k resistor (originally this is pull-up resistors for flash and enable signals for a uC, so not designed for that but anyway). The power-up settings are: all pins GPIO. This is also the state of the SRAM after boot. But then I get the following results:
After power-up
Voltage ref.: Internal (1.024V)
Pin 1: 0.00V
Pin 3: 2.76V - should be 0 since it is configured as GPIO

Set pins 1 & 3 to GPIO
Voltage ref.: Internal (1.024V)
Pin 1: 0.00V
Pin 3: 2.76V - should be 0 since it is configured as GPIO

Set voltage ref. to internal and 4.096V
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 3.26V - measured 3.29V with voltmeter, OK

Set pin 3 to ADC
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 2.75V - switched to ref = Vdd w/o being requested

Set pin 3 to GPIO
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 0.00V

Set pin 1 to ADC
Voltage ref.: Internal (4.096V)
Pin 1: 2.74V
Pin 3: 0.00V

Set voltage ref. to 4.096V
Voltage ref.: Internal (4.096V)
Pin 1: 3.26V
Pin 3: 0.00V

Set pin 3 to ADC
Voltage ref.: Internal (4.096V)
Pin 1: 2.32V
Pin 3: 2.73V

Set pin 3 to GPIO
Voltage ref.: Internal (4.096V)
Pin 1: 2.75V
Pin 3: 0.00V

Set voltage ref. to internal and 4.096V
Voltage ref.: Internal (4.096V)
Pin 1: 3.26V
Pin 3: 0.00V

Set pin 3 to ADC and pin 1 to GPIO
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 2.75V

Set voltage ref. to internal and 4.096V
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 3.26V

Set pin 1 to ADC
Voltage ref.: Internal (4.096V)
Pin 1: 2.34V
Pin 3: 2.76V - again, voltage ref. switched to Vdd

Set pins 1 & 3 to GPIO
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 0.00V

Set voltage ref. to internal and 4.096V
Voltage ref.: Internal (4.096V)
Pin 1: 0.00V
Pin 3: 0.00V

Set pin 1 to ADC
Voltage ref.: Internal (4.096V)
Pin 1: 2.75V - again, voltage ref. switched to Vdd
Pin 3: 0.00V

There's indeed something not right with the firmware.

:( I didn't find any errata regarding this behavior,
Not sure if reading SramDataSubcode.ChipSettings: and forcing to rewrite the ADC Parameters is a "good" workaround
https://github.com/vpaeder/pymcp2221/blob/master/mcp2221/__init__.py#L348-L375
.Me may use self._write(0x61)or  alter https://github.com/vpaeder/pymcp2221/blob/d8a4309a2786747de34378ec92e49e788fc8302f/mcp2221/init.py#L330 to except a SramDataSubcode.All to avoid a double read

Tried to fiddle around with this a little more. Microchip's utilities are very basic and there's no official example that'd help. I've tried with 3 other packages just to see and I get the same result. Surprising that no one complained about it on Microchip's forums.
I can't see for now how to avoid a read+rewrite but I'd stick it in https://github.com/vpaeder/pymcp2221/blob/d8a4309a2786747de34378ec92e49e788fc8302f/mcp2221/init.py#L1270 with data from self._write(0x61).

Ok I've pushed a fix, now I still have to see with the phony ADC values on startup. I don't think it affects the pin function when set to something else than ADC, but it's bothering me nonetheless.

That should be fine now. I let you check if it works on your side too.

Works fine on my side :) Thanks.
I only see that  self._read_sram get called twice:
https://github.com/vpaeder/pymcp2221/blob/b26b25fa7f447a2b39cf28c3826286d4652d6db4/mcp2221/__init__.py#L371-L375

ah yes I messed up with commit merging, oups. Fixed.


Check if conid is available to trade?

Is there a way to check if a conid is available to you except for calling marketdata_history_by_conid and see if anything is returned?

Hey @exallon72 good question! When you say 'available' - do you mean to check whether you have a market data subscription for it? I'm not aware of any method at the moment. I think you could try submitting a ticket with IBKR and see if they can suggest a solution 👍

Thanks that is what I mean.
It seems to me that this is a very big problem. So I will.

Great, let us know what they say 👍

@exallon72 did you find out an answer?


Plex_search: Genre: seems to be erroring out

Using 1.4.0, the genre: builder seems to not be working - spitting out Unknown Error: Unknown filter category: genre. Snippet from logs here: https://pastebin.com/yuwPmLnd
Config:
    Adventure:
        plex_search:
            genre:
                - Adventure
                - Action/Adventure
        trakt_list:
            - https://trakt.tv/users/retrogran/lists/adventure-movies?sort=title,asc
            - https://trakt.tv/users/niklcc/lists/action-adventure-movies

Also doesn't work with only one genre:
genre: Adventure

duplicate of #78 it should be fixed once https://github.com/pkkid/python-plexapi/pull/693 pull request is added to the PlexAPI release until then its going to, unfortunately, remain broke unless you roll back the version of your PMS


CmdDef should accept ordered parameters

What is the expected enhancement?
The command definition should accept a list of ordered parameters or kwargs.
currently, the format is:
cmd_def.get('u3', qubit, P1=P1, P2=P2, P3=P3)
It should also be possible to do
cmd_def.get('u3', qubit, P1, P2, P3)
There is a question of what the behavior should be if both args and kwargs are supplied for params, ie, cases such as:
cmd_def.get('u3', qubit, P1, P2, P3=P3)
cmd_def.get('u3', qubit, P2, P3, P1=P1)

Closed by #2456.


Administrators should be able to copy an existing project type with all barriers and questions

Questions and barriers should be duplicated to the new project type

[ ] #604

This will be replaced by #637 after discussion with costumers about using labels for sub projects instead of duplicating entire question set for each sub project type


Riya parquet

Description of changes

the code to load the data had to be modified to read the parquet files
parquet.py has a function called parquet_convert that converts the txt files to parquet and stores them in whatever folder you specify --> need to modify it to be more generic/dsmlp-friendly
the "notebooks" folder has some EDA that will be useful for later
Things to add later
generic file path

I'm on board with Emily's thoughts. Other than that, it all looks good!


Explore Electron.net API support in the utility process

It would provide us with the possibility of using chromium network stack which has better support for proxies directly from the utility process. Currently chromium transfers URLLoaderFactories to renderer process which allows blink to make network requests directly from the network service without having the need to go through the browser process, we will try to follow the same principle in this exploration.

[x] Validate the exploration using a simple prototype
[x] Scaffold Electron.net for the utility process

Prototype Implementation:
Let's create a interface on the NodeService utility process which accepts a URLLoaderFactory from the browser process and uses it to create a URLLoader to issue simple http(s) requests and outputs the response to a file.
Consists of 3 components

Declaring a mojo interface to transfer the URLLoaderFactory
Defining the interface in the Utility Process
Invoking the interface from the Browser Process

/cc @devm33

Steps for verification:

Add the following setting "github.copilot.advanced": { "debug.useElectronFetcher": true }
Reload Window
Open Github Copilot Chat output panel and check for the following log FetcherService] Using Electron fetcher
Test that chat requests are fullfilled without any network issues.


cannot inherit from non-open class Validator outside of its defining module

Hey guys,
would be nice to have the inherit even in Validator class. I need to extend some functionality. I believe that changing "public" to "open" is the magic.
Carmelo

I am having this issue in subclassing RegexRule as in the documentation example for a project in Swift 3.  Carmelo indicates what the Swift 3 docs purpose as a solution.

Please make Validator class "open" for swift 3

@smamczak Could you explain how to do this? I can not fix the code in someone else's library ...


Updated Signup Flow

Updated Signup Flow
Add new fields to signup flow

Title
Thumbnail (Avatar)
Language

Might require renaming of some labels.

Will get this up on a test server soon...

This looks good, I'll file a separate issue to update other areas to match the newer terminology (i.e. channel creation page, youtube sync flow).
Just going to add a note that usernames can't be changed.


wingetcreate update --id should prompt for the information.

Description of the new feature/enhancement
wingetcreate update --id does... nothing? if the manifest does not use a dynamic url. if -v -u -s is not inputted it should prompt for the information.
#19
Proposed technical implementation details (optional)

@OfficialEsco what do you think about removing the --id argument and requiring a "PackageIdentifier" as a positional argument?
Maybe something like wingetcreate update Microsoft.PowerToys -v <IP_ADDRESS>


OData.Feed failed to convert literal to EdmDateTimeOffset

Preflight Checklist

[X] I have installed the latest version of Power Query SDK.
[X] I have checked existing resources, including the common issues and the release notes.
[X] I have searched for similar issues.

Power Query SDK
0.2.3
Regression From
No response
Platform
Windows
Architecture
x64
OS Version
Windows_NT 10.0.19045
VSCode version
No response
PQSdkTool Path
No response
Bug Description
When pulling data using OData.Feed, the custom connector fails to convert literal '2022-08-11T00:00:00.0000000' to expected type "Edm.DateTimeOffset".
Sample code for the OData.Feed are as such:
TrainingCore = OData.Feed("url",[#"Authorization" = authToken,#"Prefer" ="odata.maxpagesize=1000000"],null),
Steps to Reproduce
1.Call OData.Feed containing field with type of 'Edm.DateTimeOffset' and string value of format '2022-08-11T00:00:00.0000000'
2.Publish the custom connector
3.Use the connector in PowerBI
4.Results
Actual Experience
After publishing the custom connector and placing the mez file to the custom connector folder, I used the connector to get data from my data source and was met with the error below:

Expected Experience
No response
Additional Context
No response

OData.Feed has two implementations internally.
You can use the [Implementation = "2.0"] option to use the latest stack. You'll likely find better compatibility with this stack.
OData.Feed("url", [#"Authorization" = authToken,#"Prefer" ="odata.maxpagesize=1000000"], [Implementation = "2.0"]),


Test for "require"

One thing we have to test (and I'm not sure it can be done through jest) is if babel is doing its shit correctly.
The way I do this locally is

npm run build
node lib

This tests if the package will be imported without errors. For example, in commit acec5d1515a0f44e5a2b5594ea57db01187de4a5 I had to add a babel library to be able to use async/await, but this wouldn't be detected just by the jest tests, I just detected it because node lib failed.
I'm not sure where we should add this test. Should it be on npm test? Or should we just add it to .travis.yml?

Fixed on 94d03072676bc4b772a17940f9af29fe8811cd18


ARROW-16939 [R] Fix nightly builds after the merge of ARROW-16407

RE2 is not available for the minimal nightly builds so this PR skips a problematic unit test.

@github-actions crossbow submit r-minmal-build

@github-actions crossbow submit r-minimal-build

@github-actions test-r-offline-minimal test-r-valgrind

@github-actions crossbow submit test-r-offline-minimal test-r-valgrind

@github-actions crossbow submit test-r-offline-minimal test-r-linux-valgrind


BadgeApp: Change format of headers

In /projects/new: Currently H1 is centered bold, H2 is italic underlined.  Don't use underline (that implies links).
Don't spend a lot of time (20 min or less), since LF may customize the bootstrap CSS settings anyway.
Perhaps use "startbootstrap.com" or other pre-existing common CSS settings.

Fixed by commit c8ef1deeaf8201115875c7411c07ef563a641475


build of xynaprod does not work

first problem is a missing
WORKDIR /opt/xyna/xyna_001/server
that leads to xynafactory.sh not being found:
=> ERROR [3/3] RUN ./xynafactory.sh start
...
/bin/sh: ./xynafactory.sh: No such file or directory
after fixing that the next problem is:
mv: cannot stat './storage/defaultHISTORY/userarchive.xml': No such file or directory

the second problem is fixed by adding a test for the existence of the userarchive.xml before moving it:
&& ( [ ! -f ./storage/defaultHISTORY/userarchive.xml ] || mv ./storage/defaultHISTORY/userarchive.xml ./storage/HISTORYwriteable/userarchive.xml )\


Refactor range

NO MERGE UNTILL this PR does : https://github.com/redcanaryco/atomic-red-team/pull/1457
This PR makes sure that attack_range and attack_range_cloud have similar functionality and code base
These commands are fully functional and tested

[x] python cloud_attack_range.py build
[x] python cloud_attack_range.py destroy
[x] python cloud_attack_range.py stop
[x] python cloud_attack_range.py resume
[x] python cloud_attack_range.py configure

TO BE DEVELOPED/Tested in a NEW PR :
StImulating AWS Attacks

[x] Added https://github.com/redcanaryco/atomic-red-team to submodules

Clone the atomic-red-team repo, since AWS simulation require the atomic test file to be present on your local machine and set the atomic_red_team_path in the conf file,

The AWS Atomic tests have not been merged yet, you will need to clone this fork for testing this PR

[ ] AWS SIMULATION PR: https://github.com/redcanaryco/atomic-red-team/pull/1457

Simulate: only simulate a one AWS TTP at a time:

[ ] python cloud_attack_range.py simulate -st T1136.003

Clean up simulation

[ ] python cloud_attack_range.py simulate -st T1136.003 --clean_up yes

dump command + wiki and README

Test and develop Kubernetes

Introduce an Attack chain file feature

@patel-bhavin do we also want to add the change to Atomic Red Team into this PR?

@patel-bhavin do we also want to add the change to Atomic Red Team into this PR?

i think we should wait on adding ART to this right away since that is also a WIP and i want feedback on that.

T

@patel-bhavin do we also want to add the change to Atomic Red Team into this PR?

i think we should wait on adding ART to this right away since that is also a WIP and i want feedback on that. but yes, that is the plan.

Then we need to disable the simulate command until ART simulation is ready. We shouldn't merge not working code into master.

@P4T12ICK : i disabled the simulate and other commands! When you get time, please test this PR!

Noticed that the  README.md still references Leonidas as well as the diagram, can we update does before merging?


enhancement: schema support for db stores

Description
Adds database support (Postgres, MySQL, SQLite) for schema storage. This PR also re-enables Admin API service endpoints for schemas.
Reason(s) to be draft;

List schemas endpoint is not done yet

Checklist

[ ] The PR title has the correct prefix
[ ] PR is linked to the corresponding issue
[ ] All commits are signed-off (git commit -s ...) to provide the DCO

Wrong branch off


publish VsCode extension

Thanks for your effort, Josh, to put this together.
I was hoping to use the VsCode extension, however, I am unable to find it on VsCode marketplace. Has the extension been published there and if not, do you have any plans getting it published soon?

The VSCode extension is not published on the Marketplace, and I don't have plans to publish it.
This language server was originally developed for a different editor, Moonshine IDE. The example VSCode extension included in the repository is mainly meant to allow me to test that the language server correctly follows the language server protocol by testing in another editors besides Moonshine.
It can also serve as an example of how someone might get started on creating a proper VSCode extension. It's really the bare minimum required to get something running VSCode, and I won't be adding any configuration options or any tighter VSCode integration. Unfortunately, I don't have the bandwidth to maintain and support a full extension for VSCode.
That being said, I'll try to find some time to update the build script so that it creates a .vsix file that people can manually install in VSCode.

Thanks for the update Josh. I would really appreciate if you can find time to update the build scripts so that the .vsix file is produced. I will then take it see if I can produce something publishable to VSCode marketplace.
Cheers!


[ISSUE #8568]feat: support get message request mode for topic and corresponding consumerGroup

Which Issue(s) This PR Fixes

Fixes #8568
Brief Description
Add a new admin interface to get message request mode for topic and corresponding consumerGroup.

Add licenses to run down the pipeline

Add licenses to run down the pipeline

done~

Resolve the conflict

Codecov Report
Attention: Patch coverage is 5.55556% with 34 lines in your changes missing coverage. Please review.

Project coverage is 47.18%. Comparing base (a1ea1eb) to head (2278123).
Report is 1 commits behind head on develop.

Files
Patch %
Lines

...ocketmq/broker/processor/AdminBrokerProcessor.java
0.00%
14 Missing :warning:

...g/protocol/header/GetMessageRequestModeHeader.java
0.00%
13 Missing :warning:

...g/apache/rocketmq/client/impl/MQClientAPIImpl.java
0.00%
7 Missing :warning:

Additional details and impacted files
@@              Coverage Diff              @@
##             develop    #8569      +/-   ##
=============================================
- Coverage      47.27%   47.18%   -0.09%
+ Complexity     11467    11455      -12
=============================================
  Files           1277     1278       +1
  Lines          89342    89378      +36
  Branches       11480    11483       +3
=============================================
- Hits           42234    42176      -58
- Misses         41918    41995      +77
- Partials        5190     5207      +17

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Remove GCP building for release promotion

Removes the dependency on GCP VMs for building Calico

make check-dirty is failing, please run make generate and commit


YAFC compatibility

https://github.com/ShadowTheAge/yafc/issues/31#issuecomment-749764783

I have added data.data_crawler for the next version
It will be equal to "yafc a.b.c.d" where a.b.c.d is the version


update dependencies

We cannot upgrade to electron v32 because electron-rebuild is failing with epoll, so staying at v31.

I will inspect
/home/bugsounet/.electron-gyp/32.0.1/include/node/v8config.h:13:2: error: #error "C++20 or later required."
   13 | #error "C++20 or later required."


Fatal error: Uncaught InstagramScraper\Exception\InstagramException: Response code is 502.

I am getting this error after to many request ...
Any idea how to by pass or fix it ..
thanks in advanced
instagram-scrapper\src\InstagramScraper\Instagram.php on line 519
line : 519
if (static::HTTP_OK !== $response->code) {
throw new InstagramException('Response code is ' . $response->code . '. Body: ' . static::getErrorBody($response->body) . ' Something went wrong. Please report issue.', $response->code);
}

I got the same problem. My script runs every 10-15 minutes, using login() or login(true) will cause my account being blocked for about 1 day. I also notice that every time I call the login() method, Instagram server will treat as a new login activity and send an email notification for new login, is it the reason for blocking my account?

Same Problem

Just change the url auth oalba:patch-1

url auth?

url auth?

Yes, just check pull requests.

Thanks!

What it will be ??
On Sat, 29 Jun 2019, 6:25 pm saki117<EMAIL_ADDRESS>wrote:

Thanks!
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/postaddictme/instagram-php-scraper/issues/506?email_source=notifications&email_token=AIHRSC6NMF66NNVPFX7VLHDP455E5A5CNFSM4HX72GWKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY32WZA#issuecomment-506964836,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIHRSC5P5FN2QXTWIRMEACTP455E5ANCNFSM4HX72GWA
.


Add redirects from the old documentation sections

Create HTML pages in each (top) folder of the old documentation structure, that will redirect to the new documentation pages. These do not show up in the documentation left-side navigation bar, and we can keep them (at least for a while) as a safety net.

Reltaed to #414

We would like an HTML page for the Check-in end-user documentation which has moved from https://docs.egi.eu/users/check-in/ to https://docs.egi.eu/users/aai/check-in/

We expect #423 fixed the errors, please let us know if it's not the case or if some URL is missing a redirection.


fix: add filtering for typescript definition files

Fix for #26 as I wrote in comment there

Closed as parsed.fileNames are the TypeScript input files and all end with .ts so the change will not filter anything. See discussion at https://github.com/ef-carbon/tspm/issues/26#issuecomment-416568240


Microsoft Excel is waiting for another application to complete an OLE action

Hello Florent,
here I am with a new issue that may be related to VBA and Selenium.
I am running Windows 7 with Excel 2010 , Firefox 42.0 and SeleniumBasic2.0.5
I am running at the same time 3 Excel instances and every instance ist using two Drivers defined as Driver and DriverB, please see below:
Public driver As New Selenium.FirefoxDriver
Public By As New By, Assert As New Selenium.Assert, Verify As New Selenium.Verify, Waiter As New Selenium.Waiter
Public Utils As New Selenium.Utils
Public keys As New Selenium.keys
Public driverB As New Selenium.FirefoxDriver
Public ByB As New By, AssertB As New Selenium.Assert, VerifyB As New Selenium.Verify, WaiterB As New Selenium.Waiter
Public UtilsB As New Selenium.Utils
Public keysB As New Selenium.keys

From the Timings I am setting:
    driver.Timeouts.ImplicitWait = 3000
    driver.Timeouts.Script = 3000
    driver.Timeouts.Server = 60000
    driver.Timeouts.PageLoad = 60000
    driverB.Timeouts.ImplicitWait = 3000
    driverB.Timeouts.Script = 3000
    driverB.Timeouts.Server = 60000
    driverB.Timeouts.PageLoad = 60000

This OLE error is happening only once per 1 Million starts of my Selenium Routine. However I tried everything that I thought could help - without success so far.
Now I am wonding if it is in general possible and stable to run Driver and DriverB in parallel at the same Excel instance.
Would be great again to get some help or ideas from you!
Thanks
Martin

Can you provide a concise and reproducible example?
How do you run/launch the code?
Parallel execution is only possible with multiple instances of Excel. Unless I'm wrong, drivers on the same instance can only be executed sequentially.

The Problem is it happening very rarely. I am starting the routines by Excel Events.
Currently I am suspecting the following:
Within Excel/VBA the OLERequestPendingTimeout is set to 5000ms and cannot be changed.
I am setting the Driver.Timeouts.Server to 60000ms.
Probably the Selenium COM library is not responding within the OLERequestPendingTimeout Limit and as I set the  Driver.Timeouts.Server to a higher time there is no error generated by the Selenium Server.
Furthermore the error is only happening when I am switching between Driver and DriverB.
As a trial I am currently changing the program to only one Driver instance and secondly I am reducing the Driver.Timeouts.Server to 4500ms.
Do you know the default setting for Driver.Timeouts.Server ?
I will keep you informed if my actions will solve the Problem.
Thanks again
Martin

The timeout for the server is set to 60000ms by default.
You can try to:

Use the last version of the driver (the waiting mechanism has been updated since v2.0.5).
Use only the driver.Wait method and not the Sleep from kernell32 which is highly io blocking
Set Application.DisplayAlerts = False when executing the script
Set Application.EnableEvents = False when writing in a spreadsheet

It could also be a deadlock issue from the two drivers competing for the same resource.

Point 3 and 4 were already integrated.
By the way one could switch off this Excel OLE alert also by the following:
Option Explicit
Private Declare Function CoRegisterMessageFilter Lib "OLE32.DLL" (ByVal lFilterIn As Long, ByRef lPreviousFilter) As Long' - See more at: http://msgroups.net/excel/excel-is-waiting-for-another-application-to-complete-ol/266667#sthash.yqrWFLHB.dpuf
'Hide OLE blocked process message box
Dim lMsgFilter As Long
CoRegisterMessageFilter 0&, lMsgFilter

However this would be my last step.
Currently I also believe that this is the deadlock issue from the two drivers competing for the same resource what you mentioned.
I never have had this issue when using only one Driver in one Excel instance with similar programs.
From next week I will test my new version with only one Driver per Excel instance, but still with up to 6 instances running in parallel.
Thanks again
Martin

Hello Florent, maybe it's a wrong place to ask questions. How can I keep browser running after VBA script is complete? Thank you.

@uzhas-sovka, your question is no related to this issue. You'll have to create a new one to expect some support.

Hello Florent,
after running in every instance only one Driver I haven't faced any problems with the OLE anymore.
However I also changed what you recommended and updated to Version 2.07 and changed the Sleep from kernell32 to Driver.wait from Selenium.
I managed to run up to 12 independet Excel instances running Selenium without problems. I also can confirm that with Driver.quit the temporary Selenium Files are deleted in the right way.
Just a small issue I saw is that Driver.url is giving the actual URL in the active Firefox tab, but Driver.baseurl is giving that one of the last opened tab.
I am not sure whether this is an issue at all - so I just want to mention it.
Thanks again for your great Support!

The .BaseUrl property holds the root of the Url used with .Get.
It makes it possible to call .Get with a short URL:
Dim driver As New FirefoxDriver

driver.Get "https://www.google.co.uk"
Debug.Assert 0

driver.Get "/intl/en/about"
Debug.Assert 0

driver.baseUrl = "https://www.google.co.uk/intl/en"
driver.Get "/policies/privacy"
Debug.Assert 0

driver.Quit


Vim airline arrows looks strange

I just installed the .vimrc file and the arrows looks strange. I have already installed the powerline fonts, I'm using Inconsolata-g for Powerline Medium 11

Any hint?

Added in your .vimrc.locals:
let g:airline_powerline_fonts = 1

@cassiobotaro Thanks!


fix(curriculum): updated the falsy condition to null check

Checklist:

[X] I have read and followed the contribution guidelines.
[X] I have read and followed the how to open a pull request guide.
[X] My pull request targets the main branch of freeCodeCamp.
[X] I have tested these changes either locally on my machine, or GitPod.

Closes #56335

Hi there,
Thanks for creating this pull request.
The linked issue has not been triaged yet, and a solution has not been agreed upon. Once the issue is open for contribution, you are welcome to update this pull request to reflect the issue consensus. Until the issue is open for contribution, we will not be able to review your pull request.


Semantic versions are not supported in scala kernel

When importing libs in a notebook, it often fails with the following message:
Ignoring unparsed versions: Vector(1.9.1+20181030090622)
This is due to a bug in Coursier that was solved in version v1.1.0-M9 (see https://github.com/coursier/coursier/issues/718).
Ammonite still uses Coursier v1.1.0-M7, so this issue will be solved once all dependencies have bumped to the right version.

Indeed, switching to the latest coursier should fix that.


try refactoring ParseBuffer for performance and readability

fixed size buffers?
byte buffers?

Done. It worked. Once again prooving that premature optimization is the root of all evil.


Bump identity.apps.core.version

Note $Subject

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/6650438355

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/6650438355
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/6653630403

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/6653630403
Status: failure

PR builder started
Link: https://github.com/wso2/product-is/actions/runs/6653665059

PR builder completed
Link: https://github.com/wso2/product-is/actions/runs/6653665059
Status: failure


all-static feature should include vendored-libgit2

Cargo has an all-static feature which is supposed to statically link dependencies for distributing.
However, it doesn't include libgit2. If the system has a compatible version of libgit2 and pkg-config installed, then the all-static build will still use the system libgit2.

@bors r+

:pushpin: Commit 1a90797c0a91e37516b5978eb66a366b0d4bce86 has been approved by ehuss
It is now in the queue for this repository.

:hourglass: Testing commit 1a90797c0a91e37516b5978eb66a366b0d4bce86 with merge 7c722f76ed702fd7fdcab449b4f5017cd48a4a6a...

:sunny: Test successful - checks-actions
Approved by: ehuss
Pushing 7c722f76ed702fd7fdcab449b4f5017cd48a4a6a to master...


Added project to OpenUPM

Dear Miha,
This is not as much an issue, but more to let you know. I have just submitted your package to OpenUPM (https://github.com/openupm/openupm/pull/4308) and it will be available shortly. With this action, people can now also install your package using openupm-cli or the openupm registry.
As you may or may not be aware, using a registry can help because Unity cannot use git repositories as dependencies in other packages; and by using an OpenUPM registered package will allow packages to list your package as a dependency.
I hope this helps, have a nice day!

And the build just succeeded, the OpenUPM page is https://openupm.com/packages/dev.krajnc.uitoolkittimeline/

@mihakrajnc something I missed when I added it to OpenUPM: there needs to be a release following SemVer (i.e. v0.1.0). Can I ask you to make a release on the releases page with a tag?
Your package.json states it is version 0.1.0, so the tag should be named v0.1.0 to make sure it gets detected properly :)

Hey @mvriel sorry for the late reply, my plan is to add it to OpenUPM but I've been busy lately, I'll take a look at all the organization / versioning the next week or two and make sure it's on there 👌


No mechanism to prevent nodes from different decentralized networks to connect to each other

I haven't yet explored source code in detais, but it seems that there is no such mechanism (at least I didn't find any). Imagine that people will start to use this library to create all sorts of decentralized things. Nodes from different swarms will connect to each other. You don't want your Bitcoin client to connect to BitTorrent swarms or different BitTorrent swarms to connect to each other. Of course this will be detected at higher level protocol but I think that Crust should detect it by itself.
I suggest that there should be some id passed to ConnectionManager which will be send to other nodes and each node will check that this id is what it wants. This id should be large enough to prevent collisions.

This is where the connect to previous known nodes works, these should all be your previous version of the network you were connected to. Your networks should not cross pollinate in any way, at least they cannot as different networks. So this should actually be OK. If no nodes are found then you can have hard coded endpoints (in the config file) to bootstrap your bootstrap cache file and then only add nodes you successfully connect to (so should be your network).
Hope that makes sense. Perhaps the docs need updated to make this very clear, I will leave this open and label docs required.

This crate had a major re-write/re-design. Please raise an issue again if required as almost everything in the library and the way it handled things have changed. Closing this.


RUN_ALL_SERVICE_TESTS is undocumented and barely used.

For some reason we have this:
https://github.com/redhat-developer/odo/blob/78a050bcf771aedeba9778e8b762c026eff7dbba/tests/e2e/service_test.go#L37
We don't use it in Travis CI, nor is there any documentation.
I believe we should remove this and run all the service tests by default. The reasoning is that running the tests locally it should automatically run ALL the tests as it won't be 100% the same as those tests being ran on Travis or other platforms.
If we decide to actually keep it, we need documentation or at least the variable be enabled by default within Makefile.

tests were refactored
/close


config file setting for bitcoin-full-nodes is ignored

first of all, is it possible to point hw at an external server?  I understand the security implications and why this might be a bad idea, I am just playing for now.
I tweaked config file for bitcoin nodes to nonsense settings to see at least this was being processed by hw, but it looks like actually it's not being processed.
How do you control the bitcoin node to connect to?
this should connect to notaserver.com:19333 (nonstandard server  with nonstandard port) but it looks like we still get default settings, no override.
[thomashartman@papaya ~/installs/haskoin-wallet]$ hw --testnet start --config=./config/config.yml
[Info] [Blockchain] Running a full block locator. This can take some time. @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:104:19)
.........
[Warn] [PeerManager] Peer 3 closed ( <IP_ADDRESS>:18333 ). Rescheduling pending jobs in the peer queue. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:242:7)
[Info] [PeerManager] Reconnecting to remote host <IP_ADDRESS>:18333 in 4 seconds. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:319:11)

tweaked settings, but no effect:
[thomashartman@papaya ~/installs/haskoin-wallet]$ git diff ./config/config.yml
diff --git a/config/config.yml b/config/config.yml
index e64d346..15970c0 100644
--- a/config/config.yml
+++ b/config/config.yml
@@ -23,11 +23,11 @@ database:
         port:     "_env:MYSQL_PORT:5432"

 ### List of trusted bitcoin full-nodes to connect to.
+ bitcoin-full-nodes:
+   - host: "test.notaserver.com"
+     port: 19333
 # bitcoin-full-nodes:
-#   - host: "bitcoin.example.org"
-#     port: 8333
-bitcoin-full-nodes:
-  - host: "_env:BITCOIN_FULL_NODE:<IP_ADDRESS>"
+#  - host: "_env:<IP_ADDRESS>:18333"

 # Log file name. Either absolute path or relative to work-dir/network
 log-file: "_env:LOG_FILE:hw.log"

Thanks for pointing this out. There may be a bug with resolving the path of the configuration file. I will look into it. However, you should be able to specify custom configuration settings by placing the configuration file in ~/.hw/config.yml. Let me know if that works for you. If it doesn't, can you send me your configuration file again? There seems to have been some display issues with your first comment.

when I place the config in ~/.hw/config.yml it works.
that leaves the problem with
-g FILE        --config=FILE            Configuration file (default: config.yml)
I was able to solve this by specifying full path
~ is not a problem for specifying a file path under home. only  the following paths cause confusion:
hw --config=config/config.yml start  # without dot
hw --config=./config/config.yml start # with dot
a quick fix that would make things somewhat clearer would be
-g FILE        --config=FILE            Configuration file (default: config.yml) # make this ~/.hw/config.yml
since the full path can be specified with --config flag.
whether it's worth fixing the confusion with current working dir, I don't know.

another wrinkle in handling ~ for homedir:
[thomashartman@papaya ~/installs/haskoin-wallet]$ hw --config ~/.hw/config.yml start # works without the =
[Info] [Blockchain] Running a full block locator. This can take some time. @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:104:19)
[Warn] [Blockchain] Trying to load an empty bloom filter @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:537:11)
[Info] [Blockchain] Rescanning merkle blocks from timestamp<PHONE_NUMBER> @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:493:15)
[Warn] [PeerManager] Peer 2 closed ( example.notaserver.com:18333:8333 ). Rescheduling pending jobs in the peer queue. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:242:7)
[Info] [PeerManager] Reconnecting to remote host example.notaserver.com:18333:8333 in 2 seconds. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:319:11)
^C
[thomashartman@papaya ~/installs/haskoin-wallet]$ hw --config ~/.hw/config.yml start # does not work without the =
[Info] [Blockchain] Running a full block locator. This can take some time. @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:104:19)
[Warn] [Blockchain] Trying to load an empty bloom filter @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:537:11)
[Info] [Blockchain] Rescanning merkle blocks from timestamp<PHONE_NUMBER> @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:493:15)
[Warn] [PeerManager] Peer 2 closed ( example.notaserver.com:18333:8333 ). Rescheduling pending jobs in the peer queue. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:242:7)
[Info] [PeerManager] Reconnecting to remote host example.notaserver.com:18333:8333 in 2 seconds. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:319:11)

. has different behavior from pwd on mac shell
[thomashartman@papaya ~/installs/haskoin-wallet]$ hw --config `pwd`/config/config.yml  start
[Info] [Blockchain] Running a full block locator. This can take some time. @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:104:19)
[Warn] [Blockchain] Trying to load an empty bloom filter @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:537:11)
[Info] [Blockchain] Rescanning merkle blocks from timestamp<PHONE_NUMBER> @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:493:15)
[Warn] [PeerManager] Peer 2 closed ( example.notaserver.com:18333:8333 ). Rescheduling pending jobs in the peer queue. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:242:7)
[Info] [PeerManager] Reconnecting to remote host example.notaserver.com:18333:8333 in 2 seconds. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:319:11)
^C
[thomashartman@papaya ~/installs/haskoin-wallet]$ hw --config ./config/config.yml  start
[Info] [Blockchain] Running a full block locator. This can take some time. @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:104:19)
[Warn] [Blockchain] Trying to load an empty bloom filter @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:537:11)
[Info] [Blockchain] Rescanning merkle blocks from timestamp<PHONE_NUMBER> @(haskoin-0.2.0:Network.Haskoin.Node.BlockChain ./Network/Haskoin/Node/BlockChain.hs:493:15)
[Warn] [PeerManager] Peer 2 closed ( <IP_ADDRESS>:8333 ). Rescheduling pending jobs in the peer queue. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:242:7)
[Info] [PeerManager] Reconnecting to remote host <IP_ADDRESS>:8333 in 2 seconds. @(haskoin-0.2.0:Network.Haskoin.Node.PeerManager ./Network/Haskoin/Node/PeerManager.hs:319:11)

I guess we should probably close this bug and re-report to the relevant file path / config arg handling library.
unless hw rolled its own solution for this.

Thanks for all this information. We can keep this bug report open for now. I wanted to refactor the configuration anyway as the current solution feels over-engineered (you can specify the configuration file and working directly at compile time and/or runtime which introduces complexity). I'll simplify all of that and at the same time drop the dependencies on Yesod (Yesod is only used for configuration stuff in haskoin).

off topic, but my next step is to actually connect to a bitcoin node.
is there any cheat for this other than running satoshi locally and downloading the blockchain?
I was looking for something to connect to on
https://en.bitcoin.it/wiki/Fallback_Nodes and bitnodes.io, but first ten addresses I checked were unreachable.
are there public servers running, that support what hw needs?
we're just downloading blocks here right? nothing that uses the json/rpc interface for satoshi client?

hw only requires a bitcoin full node (so any of the bitcoin core nodes will do). It doesn't use any of the json/rpc commands. It is basically an SPV client similar to bitcoinj. It currently fetches and validates block headers, then downloads and imports merkle blocks. We have a full node running on faucet.haskoin.com if you want to use it. Otherwise, any other full node will do.
To connect to a different node than localhost by default, you simply need to change the bitcoin-full-nodes configuration value in config.yml.

However, I'll add for anyone who may be reading that it is currently only recommended to run hw behind a trusted full node.

Paths with the ~ sign to refer to HOME are expanded by the shell only under some circumstances. One of the cases where the shell will not expand this is when there is a preceding character such as the equal sign. You may use $HOME instead of ~ in those circumstances.
About the dot character to refer to the current directory, that one does not get expanded by the shell. hw will use its working directory instead of the one you are calling it from. By default this is is .hw in HOME.

I tried haskoin faucet and I'm at 256 timeout in the exponential backoff.
I guess my next move is to see if I can find another server, or provision my own.
If you think haskoin faucet should be serving blocks let me know and I'll try again and figure out what I was doing wrong.

found a node!
I went down the list at https://getaddr.bitnodes.io/nodes/  , checked reachability using the reachability tool in previous message, and when one checked out set the config file to that address.
now downloading blocks.

As an experience report, my ad hoc node that I pulled from on from above list lasted about five minutes and then went dark.
I guess when you use the satoshi client or similar software, once it makes the first connection it bootstraps a list of more nodes that are continually refreshed, to keep the blocks coming.  And haskoin-wallet currently doesn't do that.  And that is why it is recommended to run satoshi locally, or at any rate somewhere you control.
Mostly for fun, I am experimenting with scraping bitnodes for a longer list and seeing if I get farther.
curl -sH "Accept: application/json; indent=4" https://getaddr.bitnodes.io/api/v1/nodes/leaderboard/?limit=100 | jq '.results | map (.node)' | grep -vE '[|]' | tr , "\n" | tr '"' "\n" | tr : "\n" | grep '.'
gets a hundred fresh nodes.  I guess I will be running satoshi locally soon.
It would be nice if haskoin-wallet behaved more like satoshi in this respect, bootstrapping up an eternally active list of nodes, but I guess that's not on the short list of priorities.

You can do this to your configuration file:
bitcoin-full-nodes:
  - host: testnet-seed.alexykot.me
  - host: testnet-seed.bitcoin.petertodd.org
  - host: testnet-seed.bluematt.me
  - host: testnet-seed.bitcoin.schildbach.de

For prodnet:
bitcoin-full-nodes:
  - host: seed.bitcoin.sipa.be
  - host: dnsseed.bluematt.me
  - host: dnsseed.bitcoin.dashjr.org
  - host: seed.bitcoinstats.com
  - host: bitseed.xf2.org
  - host: seed.bitcoin.jonasschnelli.ch

thanks!


Consider tracking focused Trail instead of Instance

See notes and the buggy case in https://github.com/phetsims/joist/issues/199.
Since we don't seem to be tracking any focus information on the instance itself (just storing a reference and accessing the trail and node), I'd recommend using focusedTrail instead of focusedInstance.
@samreid, I'll be available to discuss at your convenience.

I think this would be great to discuss and investigate.

I'm making good progress on trails, see the above commits.  Need to know how to listen for global bounds change of a node.  Register listeners all the way up its trail?

From Skype:

How do I tell it to focus one but not the other without having to manually construct trails?

Node.getTrails (see docs)

can I use Trail.previous/next to enumerate all focusable instances?

Trail.eachTrailBetween is better for enumerating all possible trails
Trail.previous/next would be ideal for walking the tree to find the next/previous focusable trail

Need to know how to listen for global bounds change of a node.  Register listeners all the way up its trail?

(excerpt from Skype)
Registering listeners all the way up the trail seems like the only option at this point, since the Instance's relativeTransform isn't global.

We are now using focus trails instead of instances.  @jonathanolson anything else to do for this issue?

Nothing else for this issue, thanks! Let me know if you feel I should review anything in the future (I'm closing this issue for now).


Add MultilabelJaccardEvaluator

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.00%. Comparing base (5fc6d15) to head (5e589d4).
Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##              main      #210   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files           76        77    +1
  Lines         1804      1826   +22
  Branches       174       175    +1
=========================================
+ Hits          1804      1826   +22

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


Reconnector down again

Hey HAORUAN,
Do you think someone can fix reconnector ? Is that you or someone else ?
Byebye

I won't fix it because the reason for it's not working is that the plugin has been disabled by HDT, not because there is a bug.
There is a workaround, but not sure if it's still working:
https://github.com/haoruan/HDT-Reconnector/issues/26#issuecomment-1854673769


Getting java.io.InvalidClassException: com.microsoft.kusto.spark.datasink.SinkTableCreationMode$

Hi Team,
We are using  "com.microsoft.kusto.spark.datasink.KustoSinkOptions" library in our spark scala code to write the data into adx  table. We are getting below error from last four days. Please help us to know what could be the issue for this exception.
"Caused by: java.io.InvalidClassException: com.microsoft.kusto.spark.datasink.SinkTableCreationMode$; local class incompatible: stream classdesc serialVersionUID = -8994901548193566197, local class serialVersionUID = -5675826937088959115"
Spark Code : readDf.write.format("com.microsoft.kusto.spark.datasource")
.option(KustoSinkOptions.KUSTO_CLUSTER, cluster)
.option(KustoSinkOptions.KUSTO_DATABASE, database)
.option(KustoSinkOptions.KUSTO_TABLE, temp_table)
.option(KustoSinkOptions.KUSTO_AAD_APP_ID, appId)
.option(KustoSinkOptions.KUSTO_AAD_APP_SECRET, appKey)
.option(KustoSinkOptions.KUSTO_AAD_AUTHORITY_ID, authorityId)
.option(KustoSinkOptions.KUSTO_TABLE_CREATE_OPTIONS, "CreateIfNotExist")
.mode(SaveMode.Append)
.save()
Library : com.microsoft.azure.kusto:kusto-spark_3.0_2.12:2.3.0
Thanks,
Lohith R

Please provide the details of the databricks cluster you were using (its version and such)
Usually InvalidClassException is a result of incompatible scala version.  This library version is using spark 2.4 scala 2.11.
Anyway:
Starting version 2.3.0 we introduce new artifact Ids: kusto-spark_3.x_2.12 targeting Spark 3.x and Scala 2.12 and kusto-spark_2.4_2.11 targeting Spark 2.4.x and scala 2.11. For Scala/Java applications using Maven project definitions, link your application with the artifact below in order to use the Azure Data Explorer connector for Spark.

Please provide the details of the databricks cluster you were using (its version and such)
Usually InvalidClassException is a result of incompatible scala version.  This library version is using spark 2.4 scala 2.11.
Anyway:
Starting version 2.3.0 we introduce new artifact Ids: kusto-spark_3.x_2.12 targeting Spark 3.x and Scala 2.12 and kusto-spark_2.4_2.11 targeting Spark 2.4.x and scala 2.11. For Scala/Java applications using Maven project definitions, link your application with the artifact below in order to use the Azure Data Explorer connector for Spark.

Closed for no response


Validating fields on the fly

I'm currently working with validate.js on a project but it only validates when you submit the form.
Can someone point me in the direction of what I need to get it validating individual fields when you leave them?
For example - field has an error > user enters a value and clicks out of field > validator does constraints checks
Thanks in advance!

Hey, I can see there aren't too many replies on issues around, but this looks like a core feature question that would be super helpful in many cases where live validation is needed (or at least triggering programmatically)


say that a page is a text file and link to editors

A few extra details to help noobs.

Again need to update in all the versioned docs.

Again need to update in all the versioned docs.

It's ok, we always point to the next (latest) version anyway.


Tgsmush crash with versions that share memory

Tgsmush.dll >= 2.8, the first version where the memory was shared, are crashing with effects ddraw.dll, but not with Golden ddraw.
How to trigger: any video after B5M2-C, B4M5-C, B3M6-C, B3M4-C, B2M4-C, B1M6-C and B1M5-C makes the game crash, so the missions where you start in a Gunboat by default.

I was able to reproduce the crash, there is an exception here:
https://github.com/Prof-Butts/xwa_ddraw_d3d11/blob/132a5dd6ee601906661a6253cb6f557a042b22ed/impl11/ddraw/PrimarySurface.cpp#L10428
Reading the comment above, it seems like a reoccurrence of the same bug, but in this case it is triggered by a video playback and it's not prevented by the current workaround.

Thinking of it, this seems to be related to tgsmush.dll calling Flip() just to make sure ddraw captures and updates the VR 2D overlay.
However, this causes other functions to be run and in this case an access to an invalid pointer because the CraftInstance table is not ready even after 5 frames.
The right solution is probably to create a specific function that will just update the overlay and skip anything else, preventing this type of bugs in the future.

Fixed by #73


finetuning on test set?

In stageA2_mbm_finetune.py, I found finetuning seemed to happen on test set. Am I mistaken?
https://github.com/zjc062/mind-vis/blob/7aece5706ea77b0ff254da70c14d448f81493e77/code/stageA2_mbm_finetune.py#L122
https://github.com/zjc062/mind-vis/blob/7aece5706ea77b0ff254da70c14d448f81493e77/code/stageA2_mbm_finetune.py#L137
https://github.com/zjc062/mind-vis/blob/7aece5706ea77b0ff254da70c14d448f81493e77/code/stageA2_mbm_finetune.py#L157

Hi thank you for your question!
Yes, we used only the fMRI data from the test set to finetune the encoder-decoder structure. It's a common practice in "pre-train then finetune" paradigm. There will be no data leakage as long as the paired data is not seen by the model. This method was also used in previous literature (Beily 2019).
Feel free to let us know if there are any other questions!

OK. Many thanks!


UNTIL THIS REPORT IS REMOVED - DO NOT PLAY NIGHTLIES

We are working on a massive sprite restructuring that may not be done today, but will allow bankswapping sprites as well as additional space. Stay tuned!
Last STABLE nightly before this change: https://cdn.discordapp.com/attachments/1230176045534019586/1235232711665389650/famidash.nes_29.zip?ex=66339f9b&is=66324e1b&hm=c1b061b0abc335b932e639652379dddbf7c001a9c7a659486bf382cc59dd2a3d&

Playing latest nightly seems one of those horror glitched romhacks

i originally designed the game to work with symmetrical cube sprites... if you add some that aren't symmetrical then the cube will snap back to an upright position every time it rotates 90 degrees

i originally designed the game to work with symmetrical cube sprites... if you add some that aren't symmetrical then the cube will snap back to an upright position every time it rotates 90 degrees

All part of todays fun, figuring out a new rotation routine 😝

i originally designed the game to work with symmetrical cube sprites... if you add some that aren't symmetrical then the cube will snap back to an upright position every time it rotates 90 degrees

Yeah after thinking it through this may not work as wanted, and we may be stuck with symmetrical icons.

Alex may have a solution...if this works, fuckin poggers

misclick


[Request] Rename pals

Would like to see this feature!

I see in source, attacks.json Line 171 it is "exclusive" to BlackGriffon. However, cheesed bosses also have these exclusive moves.
Recommend edits adding bosses to exclusive moves for the following pals:
Axel & Orserk
Lily & Lyleen
Marus & Faleris
Victor & Shadowbeak
Zoe & Grizzbolt

Planned for v0.9


Nuke Slate not published

Is there an existing issue for this?

[x] I have searched the existing issues and added correct labels.

Description
Current Behavior
Deadline now ignores Nuke slate, and offsets the published exr sequence.
Expected Behavior
Nuke slate should be published.
Steps To Reproduce:

make Nuke script, add Slate node, range 1001-1010
Publish on farm
Slate is extracted correctly, and added as frame 1000 in work folder
Publish ignores the slate, exr files are offseted (no  exr slate published, frame 1001 becomes 1000)

Additional context:
Version
Server 1.5.4, core 1.0.10, deadline 0.5.0, nuke 0.2.6
What platform were you running when you found the bug?
Windows 10
You can also provide any additional information relevant to the issue.
The slate exr frame file is now not present in json instances.representations.files

Relevant log output:

Will be resolved by https://github.com/ynput/ayon-deadline/pull/74

Following is perhaps missing? https://vscode.dev/github/ynput/ayon-deadline/blob/0.4.0/client/ayon_deadline/plugins/publish/nuke/submit_nuke_deadline.py#L511-L512


Update Upgrade Process to remove Telerik (Continued from #5166 and #5099)

Please see https://github.com/dnnsoftware/Dnn.Platform/pull/5099
and https://github.com/dnnsoftware/Dnn.Platform/pull/5166

After discussions on 8/2 a few more changes are planned to this:

Move and consolidate Telerik Removal code in a new library that is present upon upgrade (less code duplication, easier maintenance)
Always replace the DAM with the new RM


do i have to make my own dataset if I want to use nn.dataparallel?

❓ Questions & Help
I used to set up the dataloader like this
train = [Data(x = torch.FloatTensor(speed[:,range(i, i+seq_len*interval, interval)]),  \
                  y = torch.FloatTensor(speed[:,i+seq_len*interval+(pred_len-1)*interval:  \
                                              i+seq_len*interval+(pred_len-1)*interval+1])) \
                      for i in range(l1)]
train = DataLoader(train, batch_size=BATCH_SIZE, shuffle=False)

and train the model like this
model = Net2d(seq_len, index, weight) model.cuda() x=next(iter(train_dataloader)).x.cuda() output=model(x)
And now I want to use 2 GPUs, and I don't know how to adjust my code.
THX

According to the docs, I guess maybe I should use dataset class, but I'm not sure and I haven't figure out how

For DataParallel, you need to use the DataListLoader instead of the DataLoader, see here.

For DataParallel, you need to use the DataListLoader instead of the DataLoader, see here.

Since I'm using my own data, I think I need to define a dataset before I use DataListLoader. I used to define a list of Data, then I use DataLoader, as I mentioned before, but I think I can't use DataListLoader in the same way

You can still use a list of data objects as input. I don't see anything that should prevent you from doing that.


docs: add explanation for fsmove

Please note, the recursive behavior of the fsmove command described in this doc is not yet implemented.
This is a theoretical implementation for a more compelling use case of both the fsmove and fscopy commands requested here: https://github.com/gopasspw/gopass/issues/2202#issuecomment-1111804624

Closing this as stale. Feel free to re-open if you want to address the comments.


Upgrade cri-o to 1.21.7 for CVE-2022-0811, add patch for CVE-2022-1708

Merge Checklist

All boxes should be checked before merging the PR (just tick any boxes which don't apply to this PR)

[x] The toolchain has been rebuilt successfully (or no changes were made to it)
[x] The toolchain/worker package manifests are up-to-date
[x] Any updated packages successfully build (or no packages were changed)
[x] Packages depending on static components modified in this PR (Golang, *-static subpackages, etc.) have had their Release tag incremented.
[ ] Package tests (%check section) have been verified with RUN_CHECK=y for existing SPEC files, or added to new SPEC files
[x] All package sources are available
[x] cgmanifest files are up-to-date and sorted (./cgmanifest.json, ./toolkit/scripts/toolchain/cgmanifest.json, .github/workflows/cgmanifest.json)
[x] LICENSE-MAP files are up-to-date (./SPECS/LICENSES-AND-NOTICES/data/licenses.json, ./SPECS/LICENSES-AND-NOTICES/LICENSES-MAP.md, ./SPECS/LICENSES-AND-NOTICES/LICENSE-EXCEPTIONS.PHOTON)
[x] All source files have up-to-date hashes in the *.signatures.json files
[x] sudo make go-tidy-all and sudo make go-test-coverage pass
[x] Documentation has been updated to match any changes to the build system
[x] Ready to merge

Summary

Update to v1.27.1 cri-o to fix CVE-2022-0811
Backports patch from the CVE-2022-1708 fix onto 1.21.7
Applies PR 8712 onto Fasttrack due to high CVEs.

Change Log

cri-o updates to latest version on 1.21 branch to fix CVE-2022-0811. v1.21.6+ has fix, latest is v1.21.7
CVE-2022-1708 patch backported to 1.21.7

Does this affect the toolchain?

NO
Associated issues

#8712 (targeted main, this is being moved to fasttrack due to the impact being a high CVE score)

Links to CVEs

https://nvd.nist.gov/vuln/detail/CVE-2022-0811
https://nvd.nist.gov/vuln/detail/CVE-2022-1708

Test Methodology

Pipeline build id: xxxx

Auto cherry-pick results:

main :white_check_mark: -> https://github.com/microsoft/azurelinux/pull/8757

Auto cherry-pick pipeline run -> https://dev.azure.com/mariner-org/mariner/_build/results?buildId=548050&view=results


NumberField now bugs with default = NaN

this worked in v0.7.29.  I used the default=NaN configuration for inputs that are optional.   You can then use isnan on the associated variable to avoid extra calculations associated with the optional input.
Can NumberField be changed back so as to allow a default value of NaN and have the displayed cell remain empty?

Apologies:  It appears to have nothing to do with version number.
NumberField(1:10,default=NaN) bugs but NumberField(1:.1:10,default=NaN) initializes with a blank field.

The previous statement is now false?!  NumberField(1:.1:10, default = NaN) now bugs with the error message:
InexactError: trunc(Int64, NaN)
trunc@float.jl:805[inlined]
round@float.jl:369[inlined]
closest(::StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, ::Float64)@Builtins.jl:122
#NumberField#7@Builtins.jl:250[inlined]
top-level scope@Local: 1[inlined]
Can someone clarify what's going on?  Apologies for the changing narrative, but this is something that worked for weeks prior to today and now has unstable behavior.

I've hit the same speed bump and looked into it a bit.
It looks like the closest function in Builtins.jl has some logic to determine whether the element is outside the range (<= rmin, >= rmax) and assumes it is inside the range if not. NaN returns false on both of these accounts, but still isn't inside the range.
A simple test at the start of the function could fix this
if isnan(x); return x; end;
I've added the pull request under commit 0495293f93b65669384d75860fc2a2641a38f510
Hope this helps.

@lukeodowd Can you create a fork, a branch, and a pull request? I am not sure how to work with floating commits.

Hopefully I've done it properly this time... 213e97cebc202c2c191ca8cf22440c24bdf90edc
Thanks for your work on this


bug: trait/impl resolution no longer working with generic types

Bug Report
Cairo version:
0b0392a461ca035c896308d199681164306f3e00

Current behavior:
The following code throws the following error:
struct Vec<T> {
    items: Felt252Dict<T>,
    len: usize,
}

impl DestructVec<T, impl TDrop: Drop::<T>> of Destruct::<Vec<T>> {
    fn destruct(self: Vec<T>) nopanic {
        self.items.squash();
    }
}

Method `squash` not found on type "core::dict::Felt252Dict::<GenericParamType(quaireaux_data_structures::vec::DestructVec::T)>". Did you import the correct trait and impl?

The same problem happens with
fn vec_new<T>() -> Vec<T> {
    let items = Felt252DictTrait::<T>::new();
    Vec { items, len: 0_usize }
}

Trait has no implementation in context: core::dict::Felt252DictTrait::<GenericParamType(quaireaux_data_structures::vec::vec_new::T)>

Expected behavior:
Compilation happens successfully

Steps to reproduce:

Related code:
Run the code above.

Other information:
It used to work before I updated the compiler to the last version. I'm not entirely sure what's causing this issue, but it looks like an issue when resolving the trait methods with generic arguments.

The issue is a missing , impl TFelt252DictValue: Felt252DictValue::<T> in the generic args - as now dict prevents problematic code by adding this high level request on the value type.
we do need to understand why the diagnostic was so unclear (i found it only after doing several iterations of simplifying the code)


Integrate codeclimate

https://codeclimate.com/github/yiisoft/di

We have scrutinizer. Should be enough.


Ecto

[ ] Investigate any potential Ecto integrations for persisting
[ ] Schema?
[ ] extend existing modules
[ ] Migration generator?

Since developers would pass email or username to generate their image I do not see why we should store a path for image. A better question here is if we want to support online storage providers (like Amazon S3). Same as long as we could generate a link to S3 then we do not need storing it in database.


Document remaining commands

According to https://nim-lang.org/docs/nimsuggest.html only def, con, sug, and use are explained but according to the nimsuggest command we have a longer list (sug|con|def|use|dus|chk|mod|highlight|outline|known). It would be nice to have some explanation of these as well.

agreed but please always refer to devel docs (ie https://nim-lang.github.io/Nim/nimsuggest.html) instead of stable


Update woke to 0.8.0

Changes
Title. So we don't fall behind too much. Tested all repos and they seemed to be fine.

/assign @julz


may you upadte a AVA dataset tutorial for newbie?

🚀 Feature

NOTE: Please look at the existing list of Issues tagged with the label 'enhancement`. Only open a new issue if you do not see your feature request there.
Motivation
Your tutorial for video classification training or finetuning is just for KineticsDat, may you upadte a AVA dataset tutorial for newbie?

Pitch

NOTE: we only consider adding new features if they are useful for many users.

Hi We don't have plans to support this Pytorchvideo anytime soon. I'd encourage users to the AVA benchmarks in pyslowfast repo to do this - https://github.com/facebookresearch/slowfast


Not working with Androidx

With use androidx libraries spinner not working properly.
Can you fix it?
implementation 'androidx.appcompat:appcompat:1.0.2'

Hi @petr-stety-stetka, to use with AndroidX please ensure you have android.useAndroidX=true and android.enableJetifier=true on your gradle.properties and that your app theme extends a material component theme like Theme.MaterialComponents.Light.DarkActionBar.
If possible please also check if you are using com.google.android.material:material:1.0.0.
Cheers

In case you're using the material:1.1.0-alphaXX versions I've made changes to support them. Just update to version 1.0.1 and let me know if you still have issues.

POlease help me to resolve this
    com.tiper.MaterialSpinner materialSpinner = new com.tiper.MaterialSpinner(new androidx.appcompat.view.ContextThemeWrapper(this, R.style.Widget_MaterialComponents_TextInputLayout_FilledBox));

implementation 'com.google.android.material:material:1.2.0-alpha05'
implementation 'androidx.appcompat:appcompat:1.0.2'

Hi @rajvimal please refer to #27


fix module build failed error at Linter.parseResults

This PR contains a:

[x] bugfix
[ ] new feature
[ ] code refactor
[ ] test update
[ ] typo fix
[ ] metadata update

Motivation / Use-Case
I am using the latest eslint-loader v3.0.0 on my rejected create-react-app and encountered this issue when I tried to start the app: TypeError: Cannot read property 'forEach' of undefined. This is the fix for this error.

@buihdk fix messages of commits, CI is not passing.
type(subject?): message
try:
fix: module build failed error at Linter.parseResults

Can we add tests?

Do you want end-to-end tests or just unit tests i.e. I'll add a Linter.test.js file to cover specific cases

@evilebottnawi I don't have permission to merge :disappointed:

@ricardogobbosouza strange, you have write, what is github writes to you?

@evilebottnawi You’re not authorized to merge this pull request.

@ricardogobbosouza let's investigate this situation, i want to add permission on merge and release

@ricardogobbosouza Can you check again?

/cc @ricardogobbosouza can you give npm username?

@evilebottnawi sure, its same ricardogobbosouza

@ricardogobbosouza please enable 2FA

@evilebottnawi enabled

@ricardogobbosouza now you can do release:

npm run release
check CHANGELOG.md is good and you have new tag (npm run release do this automatically, just manual checking)
push release commit to master and push new tag
fill release (https://github.com/webpack-contrib/eslint-loader/releases), just copy/paste from CHANGELOG.md
run npm publish

@evilebottnawi thanks

@evilebottnawi @ricardogobbosouza hey man! how is it going with releasing bugfix?
P.S. I will appreciate your new release with this fix. 🙏 Actually need your attention at the moment
thanks a lot

@misha-panyushkin near future


Add CombinedRocFilter and roc-intersect-mode to vcfeval

This PR adds a flag to vcfeval called --roc-intersect-mode which toggles whether the ROC filtering conditions should be applied together ("AND") as opposed to in parallel separately (current behavior).
The Feature
An example command is the following:
rtg vcfeval -b base.vcf -c call.vcf -t ref -o out_dir \
    --roc-intersect-mode \
    --roc-regions test-reg=test1.bed \
    --roc-regions test-reg2=test2.bed \
    --roc-subset snp,non-snp,het \
    --roc-expr "stuff=INFO.TEST>5"

Normally, this command would produce 6 _roc.tsv.gz files for each of the different categories. However, when the flag --roc-intersect-mode is included you will get a ROC output for each possible combination of the filter conditions. In the subset category, two subcategories are defined internally to separate variant conditions (SNP, etc) from genotype conditions (Het, etc). In particular, each provided region, subset, and expression would be combined (along with a default "no filter" in each category) to provide (2+1 regions) x (2+1 variant types) x (1+1 genotypes) x (1+1 expressions) = 36 files in this case.
So for example, the above command would provide ROC file outputs for e.g. SNPs in test1.bed (test-reg+snp_roc.tsv.gz), non-SNP hets in test2.bed withINFO.TEST>5 (stuff+test-reg2+het+non_snp_roc.tsv.gz), and so on.
The Implementation
The code attempts to minimally change existing code, and utilizes the framework for applying ROC filters. It introduces a new class called CombinedRocFilter which is created from a list of RocFilters and holds them together. This object is itself a RocFilter, and it accepts a site exactly when it passes the accept criteria for all of the filters it holds, thus applying the intersection of all conditions it holds. Because it is a RocFilter, it can be dropped in place for where RocFilters were previously used for generating the output stat files.
A little bit of edge case reasoning is done to handle producing outputs where an arbitrary subset of filters is applied (e.g. just some of the different categories, like variant type and region, but not the others), especially the case where "no filter" is applied for global stats and backwards compatibility in the usual output. In the case the user does not toggle this flag, no CombinedRocFilters should be created, and the tool should work the same as before, applying each filter separately and generating its own 1 output file per filter.
Please let me know if you have any questions, comments, or suggestions to improve the code. Thank you!

Just pushed some minor changes to address some of the documentation changes requested above. I'll work on trying to get the tests worked out sometime soon, hopefully this week. Thanks!

Hi @Lenbok, I just added some tests as suggested for the new feature (or at least attempted to). Do you know how I can try running them to make sure they're correct? I'd also appreciate if you took a look at them to see if you're satisfied with the case coverage at some point. Thanks!

Hi @rickymagner, from the command line you can run all the unit tests via ant runalltests, and run a specific test class using (e.g) ant runtest -Dtest.class=com.rtg.vcf.eval.VcfEvalNanoTest. Depending on what IDE you use, you can probably run individual test classes/tests directly from there.
(Aside, I also just noticed you are using four space indentation, whereas we use two spaces)

Hi Len, I just addressed a few of your comments:

I switched from 4 spaces to 2 spaces for indent in the relevant files.
Fixed some of the failing filter tests.
Added a spellcheck exclusion for roc-expr which helped with the tests.

I'm still seeing two tests fail when testing locally:

Something involving the usage server, which I imagine is just specific to my machine and maybe not a general issue?

     [java] 1) test(com.rtg.usage.UsageServerCliTest)
     [java] junit.framework.AssertionFailedError: Checking usage configuration.
     [java] Clients will not be able to connect until RTG_USAGE_HOST has been set. Please consult the user manual section "Advanced Installation Configuration".
     [java] Error: Could not create socket on port 3283. Address already in use
     [java]  expected:<0> but was:<1>
     [java] 	at junit.framework.Assert.fail(Assert.java:57)
     [java] 	at junit.framework.Assert.failNotEquals(Assert.java:329)
     [java] 	at junit.framework.Assert.assertEquals(Assert.java:78)
     [java] 	at junit.framework.Assert.assertEquals(Assert.java:234)
     [java] 	at junit.framework.TestCase.assertEquals(TestCase.java:401)
     [java] 	at com.rtg.usage.UsageServerCliTest.test(UsageServerCliTest.java:98)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     [java] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [java] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
     [java] 	at junit.framework.TestCase.runTest(TestCase.java:176)
     [java] 	at junit.framework.TestCase.runBare(TestCase.java:141)
     [java] 	at junit.framework.TestResult$1.protect(TestResult.java:122)
     [java] 	at junit.framework.TestResult.runProtected(TestResult.java:142)
     [java] 	at junit.framework.TestResult.run(TestResult.java:125)
     [java] 	at junit.framework.TestCase.run(TestCase.java:129)
     [java] 	at junit.framework.TestSuite.runTest(TestSuite.java:252)
     [java] 	at junit.framework.TestSuite.run(TestSuite.java:247)
     [java] 	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:128)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:27)
     [java] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
     [java] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
     [java] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
     [java] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
     [java] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
     [java] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:128)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:27)
     [java] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
     [java] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
     [java] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
     [java] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
     [java] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
     [java] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:105)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:94)
     [java] 	at RtgTestEntry.main(RtgTestEntry.java:175)

A problem with VcfEvalNanoTest.

     [java] 2) testNanoRocCrossJoin(com.rtg.vcf.eval.VcfEvalNanoTest)
     [java] java.io.FileNotFoundException: /var/folders/w5/8rg1mj0j1rx9skg02s9s_r900000gp/T/unitTest7985174708251715361vcfeval-nano/output/snp_roc.tsv.gz (No such file or directory)
     [java] 	at java.base/java.io.FileInputStream.open0(Native Method)
     [java] 	at java.base/java.io.FileInputStream.open(FileInputStream.java:219)
     [java] 	at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
     [java] 	at java.base/java.io.FileReader.<init>(FileReader.java:75)
     [java] 	at com.rtg.util.io.FileUtils.fileToString(FileUtils.java:639)
     [java] 	at com.rtg.launcher.AbstractEndToEndTest.checkResultFile(AbstractEndToEndTest.java:91)
     [java] 	at com.rtg.launcher.AbstractEndToEndTest.lambda$endToEnd$0(AbstractEndToEndTest.java:56)
     [java] 	at com.rtg.launcher.AbstractEndToEndTest.checkResults(AbstractEndToEndTest.java:85)
     [java] 	at com.rtg.vcf.eval.AbstractVcfEvalTest.endToEnd(AbstractVcfEvalTest.java:73)
     [java] 	at com.rtg.launcher.AbstractEndToEndTest.endToEnd(AbstractEndToEndTest.java:54)
     [java] 	at com.rtg.launcher.AbstractEndToEndTest.endToEnd(AbstractEndToEndTest.java:50)
     [java] 	at com.rtg.vcf.eval.VcfEvalNanoTest.testNanoRocCrossJoin(VcfEvalNanoTest.java:164)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     [java] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     [java] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     [java] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
     [java] 	at junit.framework.TestCase.runTest(TestCase.java:176)
     [java] 	at junit.framework.TestCase.runBare(TestCase.java:141)
     [java] 	at junit.framework.TestResult$1.protect(TestResult.java:122)
     [java] 	at junit.framework.TestResult.runProtected(TestResult.java:142)
     [java] 	at junit.framework.TestResult.run(TestResult.java:125)
     [java] 	at junit.framework.TestCase.run(TestCase.java:129)
     [java] 	at junit.framework.TestSuite.runTest(TestSuite.java:252)
     [java] 	at junit.framework.TestSuite.run(TestSuite.java:247)
     [java] 	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:86)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:128)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:27)
     [java] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
     [java] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
     [java] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
     [java] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
     [java] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
     [java] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:128)
     [java] 	at org.junit.runners.Suite.runChild(Suite.java:27)
     [java] 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
     [java] 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
     [java] 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
     [java] 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
     [java] 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
     [java] 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:105)
     [java] 	at org.junit.runner.JUnitCore.run(JUnitCore.java:94)
     [java] 	at RtgTestEntry.main(RtgTestEntry.java:175)

Here it cannot compare the files properly. The way it's currently written, the command outputs *_roc.tsv.gz files which are compressed, but I committed non-compressed files. The issue is that two gzipped files can't be compared directly because there is often metadata like timestamps included in their headers. The way the testing infrastructure seems to be set up, it only allows comparing file strings directly to assert they are equal.
But even compressing my test output files doesn't seem to help. I get the same stacktrace saying it cannot find the right files.
What do you think is the best way to resolve this? I'm not so sure it's as much a matter about comparing .gz files (though suspect that might eventually be an issue if not this one) or rather I am not calling the function with the right paths.

Hi @rickymagner,
Regarding the usage server test, I'm not sure what is causing that - the error message indicates something on your machine might already be listening on port 3283?
Regarding VcfEvalNanoTest though, the endToEnd test adds on the -Z flag for uncompressed output files, so you need to remove the .gz from the files you're listing that need to be checked. The next thing is that during regression results comparison, the files undergo simple sanitization to remove version numbers and command lines, but the expected outputs you've committed still have those present.
If you run the test with ant runtest -Dtest.class=com.rtg.vcf.eval.VcfEvalNanoTest '-Dregression.update=.*' the nanoregression will update those expected outputs (and you'll be able to see from git diff that it's just those header lines that change).

Great, it looks like that worked to get all tests passing! Not sure what was listening on that port before, but after a restart all the tests passed for me. I just committed those small version updates.
Please let me know if there's anything else I can do to help this get merged! Thanks for all your help so far.

Hi @Lenbok, thanks for the reply. I agree with your first point that it'd be nice to make it clear that we'd be considering SNP vs INDEL as an orthogonal choice to HET vs HOM, when they were previously considered on the same footing with that flag. Would you suggest actually replacing the --roc-subset flag entirely with those two? I'd be in favor of it from a technical standpoint, but worry about breaking backwards compatibility.
For the second point, I agree it could be a little easier to work with having one output ROC rather than a possibly large collection (and then having to remember the inputs to the command to grab them all up into one table, or at least that's what I planned to do). If I'm understanding correctly, the flag --roc-subset-mode would be new and have the behavior:

default: the current behavior
intersection: combine orthogonal filters together "multiplicatively" rather than the current "additive" behavior, i.e. snp + interval would just counts SNPs over that interval
cross-join: the behavior of this PR (but maybe combined into just one ROC? How would that look? Some extra column(s) to designate the choice for that row?)

I think the second option sounds more appealing from the user perspective, although would require a bit more work. I'm OK with potentially trying to get a flag like that implemented if you agree since it shouldn't be much harder than what's already been done here. I'd prefer to try to get things done "right" before modifying the pipelines I want to include it in so those won't have to be changed so quickly after.

For the first point I would definitely not want to break backward compatibility either. We don't necessarily have to do something about this, it's just less than optimal.
For the second you have the right interpretation. Although, now that I think of it, intersection is just a special case of cross-join where none of the "orthogonal filters" had more than one value selected. So maybe we don't need to leave room for three modes.

It's possible for the first point we could deprecate the old flag but leave it in for now, and print a warning to use the two new flags instead going forward?
That's a good point that the intersection is a special case. So that means the mode is completely controlled by the presence/lack of the --roc-cross-join flag then?
I think there is a use case in flexibility there. I agree for the het/hom vs snp/indel, have het_snp, hom_snp, and snp could be excessive since the latter is already captured by the first two (and can be deduced algebraically from the first presumably). But in the case of restriction to a bed region, you might want both mybed+snp and snp. Now the latter is not possible to deduce from the former.
Perhaps there can be some flexibility in how the user wants these to be handled by actually allowing them to specify when the ALL filter should be added there by doing something like:
... --roc-cross-join --roc-subset snp --roc-regions "mybed=mybed.bed" --roc-regions ALL

which would produce mybed+snp and also ALL+snp = snp (and would be printed as the latter).
Now ALL would be a sort of keyword argument that means "also do the cross product where this value has no restriction added to the ROC computations." What do you think? Is it too confusing?

OK, good discussion. Here's what I think we should do:

For the flags, lets leave them as they are currently, and when I update the documentation to describe the new flag I'll talk about the notional categories within --roc-subset as being gt vs variant related.

For the four per-category RocFilter.ALLs, I think that for now we should keep things simple by just adding them when the category is otherwise empty. It'll keep the number of output files managable and it's forward compatible with your suggestion of allowing special ALL command-line keyword (although we'd probably want to do the splitting out new flags for gt vs variant subsets at that time).

(BTW, are you aware of rtg vcf2rocplot, which makes it fairly cheap to generate new ROCs after the primary rtg vcfeval run as long as you output annotated VCFs?)

Sounds good on the flags. For the ALLs, I think I agree that generally it probably doesn't make sense to always add it for each category (like the ones I mentioned before, e.g. Het/Hom). I suppose if I want to also get a "whole genome" default interval value, I can always input a whole genome bed file to cover that case.
I wasn't aware of vcf2rocplot, but it sounds like that would be subject to the same issues of trying to combine multiple filters at once, e.g. getting het+snp wouldn't currently be possible for the same reasons vcfeval can't (which this PR addresses). It's possible that tool could get a similar upgrade, but is probably beyond the scope of this PR.
So it sounds like the only thing left to do on my end would be: change the way the filters are collected by only adding an ALL if the user provided no filters for each category, correct? I could probably get that done early next week and update the tests if needed so it'll be ready to merge.

@rickymagner I just realized that as it stands this will actually break rtg vcf2rocplot and rtg cnveval (they both make use of the VcfEvalCli functions for roc filter setup). Obviously they don't have unit tests for this roc subsetting otherwise there'd be a test failure. Essentially rtg cnveval defines (only) two options for --roc-subset, del and dup. The new VcfEvalCli.getRocFilters will drop those guys on the floor. rtg vcf2rocplot is intended to allow processing either vcfeval or cnveval outputs, so it allows the union of the cnveval and vcfeval roc subsets (and simililarly the cnv roc subset processing will no longer work).
Can we add an extra vararg parameter to getRocFilters, like
  public static Set<RocFilter> getRocFilters(CFlags flags, List<RocFilter> fallbackRocFilters, Set<String>... rocSubsetGroups) throws IOException {

where vcfeval would call it passing in two two sets (the same as you've currently got internally as genotypeOptions and variantOptions), cnveval would call it with one set containing the del and dup filter names, and vcf2rocplot would call it with three sets. The innards of getRocFilters would need to handle the variable number of subset groups.

Ah I didn't know cnveval exists as it's not in the RTG manual, but sure enough it has a --help output on the command line. I think your suggestion makes sense to add another input to the method so different tools can configure how to call it with rocSubsetGroups varying. Can you say a little more about what values vcf2rocplot should call the method with? I can try to make these changes hopefully sometime this week.

cnveval is a fairly early stage tool that we are planning to promote in the next release, which is why it doesn't appear in the manual or output of rtg --help. I agree regarding pulling our that shared functionality at some point in the future.
vcf2rocplot could either supply three different rocSubsetGroups (One for the cnveval subsets, and two corresponding to the vcfeval subsets for genotype and variant type), or alternatively you might consider the cnveval subsets del and dup as just being additional members of the variant type subset.

Nice, good to know. Do you think it's still possible this PR would be able to get into a minor release before whenever you're thinking of that next major release including that tool?
So to be clear for the refactoring of getRocFilters, I should:

Add the new rocSubsetGroups input to the function, which would be a set containing elements like snp, non-snp, indel, etc... for vcfeval and del, dup for cnveval.  Do you want the method to also check consistency of these choices with the tool? If there are filters for del and dup which make sense as DEL/INS for vcfeval, it could be an addition of new options for the tool and handled the same way, no?
Remove the addition of RocFilter.ALL to each category in the roc-cross-join mode unless no filtering conditions are given for that category.

Did I miss anything else? I think we might want a test or two for the vcf2rocplot and cnveval, right? Although I'm not adding the cross join functionality to them, so perhaps it doesn't apply?

Nice, good to know. Do you think it's still possible this PR would be able to get into a minor release before whenever you're thinking of that next major release including that tool?

I'm thinking we'd do it as one release, but we would try and do it promptly.

* Add the new `rocSubsetGroups` input to the function, which would be a set containing elements like `snp, non-snp, indel, etc...` for `vcfeval` and `del, dup` for `cnveval`.  Do you want the method to also check consistency of these choices with the tool? If there are filters for `del` and `dup` which make sense as DEL/INS for `vcfeval`, it could be an addition of new options for the tool and handled the same way, no?

I think a sufficient consistency check is that when filtering through the user-selected roc subsets, they should always fall into one of the rocSubsetGroups, and raise NoTalkbackSlimException otherwise. I wouldn't change the set of subset options currently available to the tools.

* Remove the addition of `RocFilter.ALL` to each category in the `roc-cross-join` mode unless no filtering conditions are given for that category.

Correct.

Did I miss anything else? I think we might want a test or two for the vcf2rocplot and cnveval, right? Although I'm not adding the cross join functionality to them, so perhaps it doesn't apply?

By making it part of getRocRilters you have added that functionality to them :-).
You can probably just update/copy the existing tests in Vcf2RocplotTest and CnvEvalTest for these to check for expected output files, and possibly again when adding --roc-cross-join.

Hi @Lenbok, I took a stab at refactoring getRocFilters to follow the scheme outlined above. I updated the vcfeval test and it was all passing on my machine. Let me know what you think of the latest commit when you get a chance. I might've misunderstood something, but did you also want me to change anything in the cnveval / roc2plot tests? Or would those still be fine if the old tests are passing?
On a side note, I added the NoTalkbackSlimException in the case the getRocFilters groups doesn't fit the set for the tool. I was curious about what the "talkback" stuff is in the tool kit. It seems like a way to report errors automatically to a database somewhere.  I was worried when experimenting earlier that I might've been spamming some database with errors that were just me debugging some changes. I also wonder if some people might be nervous about sending sensitive filenames/paths in the log unknowingly if using the tools.

Hi @rickymagner I added a couple more commits to this PR, take a look.
I also tweaked the current cnveval unit test to use roc subsets, and they continue to pass. I also experimented with adding the --roc-cross-join flag to that unit test, it has the effect of enabling rescaling on the del/dup roc files. (In this PR the CombinedRocFilter has rescaling enabled when there were no --roc-expr filters). We might instead want to set rescale to null, so that it uses the global default, but I'm curious what you expect the behaviour should be when using cross join mode.
Regarding the crash reporting, the setting for that is in the rtg.cfg file that gets created when you run rtg for the first time. Crash reporting is opt-out (there's a separate opt-in usage logger that excludes command lines for seeing how often the different subcommands are used).

Hi Len, thanks for the extra commits. I think it's a good idea to refactor to make the combined roc filters separately, and thanks for adding some extra docs. For the rescaling: I think I just did whatever seemed to "make sense" based on some experimenting, or maybe an error was thrown when the expr filters didn't have it set. I don't have any expectations on what "should" happen here necessarily, so would defer to whatever you think is appropriate. IIRC, I think it only has a small effect on the outputs, right?
For the reporting, now that I think about it I don't think I've ever had an issue with the program hanging in a docker image. I suppose a default cfg file must be used in that case where you aren't running interactively?

Hi Ricky,
I've adjusted the logic for whether to enable rescaling. Now it requires all the sub-filters to have rescaling enabled in order to turn on. I think this better matches the principle of least surprise, in that using --roc-cross-join when there is only a single "dimension" will not alter the use of rescaling. If you're using multiple dimensions it'll turn off rescaling if any of them have rescaling turned of, and the rocs will then have raw numbers.
I think this looks OK now, so if you're good with it, I'll do a little commit squashing and merge.

BTW, thanks for the contribution! We'll work on getting a new release out.

Thanks, and thanks for all your help! Looking forward to the formal release to incorporate into our pipeline.


Please use forward slashes in examples

Please use forward slashes in examples, otherwise linux build agents don't match in Azure DevOps.
As per examples in https://docs.microsoft.com/en-gb/azure/devops/pipelines/tasks/file-matching-patterns?view=azure-devops

Thank you for reporting this issue. It's been fixed.