Add support for string ids for COCO dataset

The loadImgs and getAnnIds in pycocotools expect a list of ids or single int id. They will treat single string id as list of ids. Always passing a list will work.

Thanks for fixing it!


MINOR: fix compile error

We use internal APIs that got changed upstream. Need to update the code
accordingly.

I also have a PR here: https://github.com/confluentinc/kafka-streams-examples/pull/400, I think we need to add the fix to InventoryServiceTest

Closing this PR, as there is already https://github.com/confluentinc/kafka-streams-examples/pull/400


[Bug]: Use UUID as identifiers

batch name and line
In all identifiers.
For example here: https://github.com/Inria-Empenn/BEP028_BIDSprov/blob/master/results/nidmresults-examples_spm_2_t_test_batch.jsonld#L26
What happened?
The identifier is not a UUID : "@id": "niiri:cfg_basicio.file_dir.file_ops.file_move._1gWIqVrGmQZ".
Describe the expected ground-truth ?
The suffix (in the above example cfg_basicio.file_dir.file_ops.file_move._1gWIqVrGmQZ) should be replaced by a random identifier generated using the uuid library. This would lead to something like:
"@id": "niiri:08f5a59a47f33372cedc55d11fc67e1a"
(Note niiri will also be replaced by urn as described in #6)
What soft is concerned
No response

Fixed in the spm_fsl_exporter branch. All elements in @id are composed of "urn:" and  a "uuid" (v4)


Add cucim OpenSlideIO API support to the CLI

This will possibly give us an all GPU pipeline from read to write with the cucim anti-aliasing and kvikio in #68.
CC @gigony

Also a flag to enable a dask-cuda cluster.


Updated Readme

Guidance for what to do when stuck.

Actually, on this particular repository, if you post an issue I'll probably ignore it and let you sort it out.


Recall Spell

Hi, I recently installed your plugin onto my server, and realised that the recall spell doesnt work, there is no list of places to go. How would i add places to that list?

You have to customize the spell, how would it know where you want it to go otherwise?
It can work with warps from Essentials or CommandBook.  See here for an example of how to customize it:
https://github.com/elBukkit/MagicPlugin/blob/master/src/examples/elpotter/spells.yml#L10
The "warps" block specifies warps, the "commands" block specifies commands to be run.


Simplify backend by storing enrollments instead of assignments.

Problem:
At present we are storing the assignments for the experiment user and to get the enrollments we need to join the table of assignments and monitored experiment point. From the application perspective if we store enrollments instead of assignments we can simplify many queries. Another problem which we are facing is after the enrollment complete state if the user reaches the experiment site they are counted as enrolled because we already have an entry in assignment table.
Changes in table
IndividualAssignmnet -> IndividualEnrollment
GroupAssignment -> GroupEnrollment
Instead of storing the enrollment per experiment we will be storing it per experiment site so that the new design can be used for factorial experiment also.
Changes in service

Now getAllExperimentCondition will not add any entry in the database (IndividualEnrollment or GroupEnrollment). It will return the assignments using the case code as we have now but for new assignment it will use random seed.
Individual Experiment -> RandomSeed will be experimentId_userId
Group Experiment -> RandomSeed will be experimentId_workingGroupId (based on type group experiment is defined on).
monitorExperimentPoint will add the entry inside the Individual Enrollment table and Group Enrollment table in addition to what it is doing at present. Also if the experiment is not in enrolling state it will not add any entry in the enrollment tables.
Update queries in the codebase accordingly.

This feels like a pretty major tech debt initiative, but I would hope to see it prioritized above new features, it would make a huge difference.


Methods for manipulating and querying tokens

We should be able to turn Token into a monad, the easiest way to do it would be to implement IntoIterator<Item = &Token> (and Iter) on &Token. It would also be great to have methods like first(&self, &str) -> Option<Token>, children_named(&self, &'static str) -> impl Iterator<Item = &Token> and all_children(&self, &str) -> Iterator<Item = Token>.

[X] IntoIterator
[ ] all_children
[ ] named_children
[ ] first


Fix misconfiguration

Hi, I found two code mistakes.

The default value of the seek position has changed.
Therefore, it can be played only once.
https://github.com/luanpotter/audioplayers/blob/98281f612f42b351b5cfe94f5dc66e24026d801e/darwin/Classes/SwiftAudioplayersPlugin.swift#L237
https://github.com/luanpotter/audioplayers/blob/cda8c39a01a0e6a0cd53f9c9ba515efc1c518c05/darwin/Classes/AudioplayersPlugin.m#L171

The session is not deactivated.
https://github.com/luanpotter/audioplayers/pull/710/files#diff-f48d6c32c3c16cb98ecc2514598b0c3eeff787b78ff213b2aa5960ee647c384cL346-L361
     func maybeDeactivateAudioSession() {
         let hasPlaying = players.values.contains { player in player.isPlaying }
         if !hasPlaying {
-            setAudioSessionActive(active: false)
-        }
-    }
-
-    func setAudioSessionActive(active: Bool) {
-        #if os(iOS)
-        do {
-            try AVAudioSession.sharedInstance().setActive(active)
-        } catch {
-            log("Error inactivating audio session %@", error)
+            configureAudioSession(active: true)
         }
-        #endif
     }

https://github.com/luanpotter/audioplayers/blob/cda8c39a01a0e6a0cd53f9c9ba515efc1c518c05/darwin/Classes/AudioplayersPlugin.m#L838
  for (NSDictionary *playerInfo in [players allValues]) {
    if ([playerInfo[@"isPlaying"] boolValue]) {
      hasPlaying = YES;
      break;
    }
  }

  if (!hasPlaying) {
    NSError *error = nil;
    [[AVAudioSession sharedInstance] setActive:NO error:&error];
  }

This change has affected since v0.17.0.
Maybe, this pr is related #671.

Thanks!


Update certificate.md

keyAlgorithm value should be lowercase
Signed-off-by: Michael Gregson<EMAIL_ADDRESS>

Thanks for that :smile:
/lgtm
/approve
/ok-to-test


Fix non-contiguous flow field and added EDT helper func

Codecov Report
Attention: Patch coverage is 90.90909% with 1 lines in your changes are missing coverage. Please review.

:exclamation: No coverage uploaded for pull request base (main@392d75a). Click here to learn what that means.

Files
Patch %
Lines

ultrack/imgproc/segmentation.py
88.88%
1 Missing :warning:

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #79   +/-   ##
=======================================
  Coverage        ?   85.45%
=======================================
  Files           ?       71
  Lines           ?     3911
  Branches        ?        0
=======================================
  Hits            ?     3342
  Misses          ?      569
  Partials        ?        0

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Question]: Properly using the -o --output switch (Permissions Error)

What is your question?
Is there any sort of permissions or something I am missing that would cause the -output function to not work. No matter what I try it throws permissions error on writing file. Here is an example command I am running:
sudo echo "extract high and critical items from aquasecurity container vulnerability scanning results and present in a clean easy to read format with short executive summary in less than 10 words for each high and critical item" | fabric --pattern create_pattern -o system.md && sudo mv container_vulnerabilities/system.md ../.config/fabric/patterns && cat ../results.txt | fabric --stream --pattern container_vulnerabilities
(error creating file: open system.md: permission denied)
No rush and thanks a TON!

Works for me beside the pattern container_vulnerabilities does not exists. What is current working directory? Maybe a permission problem, as in my understanding sudo only works for echo not for fabric in your command.

Running from /home/jnuts74
So I broke this down a bit to continue testing to get the output function working before trying to string together execution of other commands. Here is where I am today:
echo "ability to perform network analysis to identify both security and network performance issues based on examining packet captures" | fabric --pattern create_pattern -o system.md
Error: error creating file: open system.md: permission denied
I even did the unthinkable and 777 my home directory where Fabric is writing the output to and still get permissions error.
Definitely strange.

Does the following work?
echo "test" | ./fabric > foo.txt

Does the following work?
echo "test" | ./fabric > foo.txt

This does work.

ok, well you have a workaround, but I'm still interested in figuring out what is occurring.
what about the following test?

echo "test" | ./fabric -o foo.txt

ok, well you have a workaround, but I'm still interested in figuring out what is occurring. what about the following test?

echo "test" | ./fabric -o foo.txt

So that also works:
echo "test" | fabric -o foo.txt
Of course! If you have any questions or need assistance with something, feel free to let me know.
... written to foo.txt
I messed with it a bit more and was able to get this absurd command to work:
sudo mkdir -m 777 summarize_container_vulnerabilities && echo "extract high and critical items from aquasecurity container vulnerability scanning results and present in a clean easy to read format with short executive summary in less than 10 words for each high and critical item" | fabric --pattern create_pattern -o summarize_container_vulnerabilities/system.md && sudo mv summarize_container_vulnerabilities .config/fabric/patterns && sudo chmod -R 755 .config/fabric/patterns/summarize_container_vulnerabilities
It appears that fabric doesnt have the ability to output to directory/file (pattern_name/system.md), so the only option is to create the directory upfront. Issue then is that since you created the directory manually, fabric  doesnt have permissions to write to it.
I am wondering if its worth putting in feature request to make fabric output to create directory and file :
fabric -o directory/system.md
Thoughts?

That's unusual use case, to use software to write to it's own folders.
Unless you're making a lot of patterns in this way, I would think manually positioning it after testing would be the way to go.
During testing save to a file somewhere like this
foo | ./fabric --patter foo -o /mytestfolder/summarize_container_vulnerabilities.md
the test using the adhoc pattern method (pattern file can be anywhere)
foo | fabric --pattern  /mytestfolder/summarize_container_vulnerabilities.md
as long as the pattern name begins with ~/ or ./ or ../ or /  it will treat it as a adhoc pattern filename.

Found slight work around but decision made to manually move newly created patterns to appropriate location as suggested by @mattjoyce
Thank you


sigmoid focal loss

The idea is to support a sigmoid focal loss, as presented in
https://arxiv.org/pdf/1708.02002.pdf
and implemented in
https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/focal_loss.py .
Porting the function was possible because all necessary components already exist in Optax.
As discussed initially at https://github.com/google-deepmind/optax/pull/574#issuecomment-1754637077,
a revert of the changes to resolve the conflict closed the request, so I am re-opening it here.
I don't know if a segmentation module is the best place possible. It could also be in classification since we are classifying pixels, but I fear new users would find that confusing, so I am thinking a segmentation module might be appropriate.

Converted to draft mode, for now. I would like to add test cases.

I think it's ready. If you have feedback, please let me know.

there are some conflicts with master. Could you resolve these before merging @v0lta ? Thanks!

there are some conflicts with master. Could you resolve these before merging @v0lta ? Thanks!

One test says Waiting for internal safe review approval, I don't think I can fix this.

Great, thanks
On Tue, Dec 19, 2023, 17:33 Moritz Wolter @.***> wrote:

@.**** commented on this pull request.
In optax/losses/_classification.py
https://github.com/google-deepmind/optax/pull/599#discussion_r1431666994
:

gamma: Exponent of the modulating factor (1 - p_t) to

 balance easy vs hard examples.

Returns:
A loss value array with a shape identical to the logits and target
arrays.
"""
chex.assert_type([logits], float)
labels = labels.astype(logits.dtype)

see also the original paper's implementation at:

https://github.com/facebookresearch/fvcore/blob/main/fvcore/nn/focal_loss.py

p = jax.nn.sigmoid(logits)
ce_loss = sigmoid_binary_cross_entropy(logits, labels)
p_t = p * labels + (1 - p) * (1 - labels)
loss = ce_loss * ((1 - p_t) ** gamma)
if alpha:

Sure, I think @chex.all_variants takes care of jit, too. The problem was
that the default value of alpha never changed. 74d17f0
https://github.com/google-deepmind/optax/pull/599/commits/74d17f0a76113d7dedd46f8e750064b4c0651c97
fixes this issue.
—
Reply to this email directly, view it on GitHub
https://github.com/google-deepmind/optax/pull/599#discussion_r1431666994,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AACDZB5EWEBJPIOMDWN6B6TYKG6XJAVCNFSM6AAAAAA54HAFFSVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTOOBZGI4DKNBQGU
.
You are receiving this because you commented.Message ID:
@.***>

Thanks for all the work @v0lta . I'm now just waiting for the internal review, which should be done within the next days

This has now been merged. Thanks @v0lta and apologies for the slow review on this one!
Note: I made some cosmetic changes when submitting due to internal errors. Feel free to open another pull request if something is missing!


Fix GP noncanonical contract

This PR contains several fixes to the noncanonical contract.

In the generator I create a deepcopy of the operands when inserting instrumentation. Otherwise further instrumentation that modifies the instrumented instruction's operand also modifies the instrumentation instructions ' operands.
In the noncanonical contract, during handling of the GP exception, we overwrite the register that contains the address. We need to restore its content afterward.
In the noncanonical address pass  we prevent registers reuse. This makes the contract implementation logic simpler. We  avoid cases where the destination is partially overwritten and should be properly restored after the exception.

@OleksiiOleksenko this is ready for review


Latest version of mp4parser

In my project I am also using the latest version of mp4parser:

implementation 'com.googlecode.mp4parser:isoparser:1.1.21'

Which making the generated video has no sound. How to fix this please?

I'm also using the 1.1.21 version of isoparser in my project and when I build my project, the gradle gives an error: 'Duplicate zip error'. Because I have two isoparser library in my project (one that I myself defined and one for silicompressor). but I need both of them. So how to handle this problem?


hv_vline~

Doesn't work with [0, 1 $1(, requires [0 0, 1 $1( instead.

Output envelope is different from [vline~] and can substantially change the sound when very fast envelopes are required for things like granulary synthesis.

hi @diemildefreude

Doesn't work with [0, 1 $1(, requires [0 0, 1 $1( instead.

This is one of the main reasons the [hv.vline] implementation isn't used as a drop in replacement for [vline~] in heavy because it doesn't exactly replicate the same behaviour in Pd. It originated as a basic replacement for sequencing envelope points in an easier way.
Feel free to submit a PR that improves on the parity with Pd though!

Output envelope is different from [vline~] and can substantially change the sound when very fast envelopes are required for things like granulary synthesis.

Again see above. Although also in this situation I would recommend using [phasor~] for your grain envelopes instead.

Hi Joe, I've just tried to replicate vline's behavior (in this situation, i.e. resetting to 0 and going back up to 1) using phasor~ by feeding it the desired envelope length in milliseconds. I'd upload it, but for some reason, GitHub is telling me it doesn't support that file type, even though it's a .zip file (made with WinRAR). I can e-mail it to you, but here's the layout:
[1000(
|
[t b    f      b         f         f]
|        |       |          |        |
[delay]      [1000] |       |
|               |         /       |
[0(           [/       ]        |
|           /                 /
[sig~]                 /
|                   /
[phasor~]
The value stops short at .99... and it sounds a bit less smooth than the vline~. It's closer than the Heavy abstraction comes though.


s and k bug in regex search

Does this issue occur when all extensions are disabled?: Yes

VS Code Version: 1.54.3 (system setup)
OS Version: win10

Steps to Reproduce:
See image:

\u017f is ſ.
k and K have similar bug as well.

Temp walk around:

We actually convert this to a JS regex i.e.:
/\u017f/giu.test('s')

And it looks that's how the JS regexp engine behaves.

We actually convert this to a JS regex i.e.:
/\u017f/giu.test('s')

And it looks that's how the JS regexp engine behaves.

Thx. So it looks like by design. ſ is s in latin.
Should I close it? @alexdima

:+1: Let's close, there's nothing we can really do on our side.


Added AxisSpec file for Axis conversion tests

Fix for #5233.

Thanks for the reviews, @lilleyse! The file has been updated.

Thanks @moneimne!


license confusion

Some of the recently added modules have unclear licensing. For instance:

rdkit/Chem/EnumerateHeterocycles.py is copyright DE Shaw Research (including "all rights reserved") without any reference to the BSD governing the rest of RDKit.
Contrib/Fastcluster and Contrib/IFG have no license information, but do mention specific authors

I can ask Brian Cole to update the code.   This is our fault for not checking the license before being accepted.

Brian Kelley

On Jun 22, 2018, at 9:13 PM, Steven Kearnes<EMAIL_ADDRESS>wrote:
Some of the recently added modules have unclear licensing. For instance:
rdkit/Chem/EnumerateHeterocycles.py is copyright DE Shaw Research (including "all rights reserved") without any reference to the BSD governing the rest of RDKit.
Contrib/Fastcluster and Contrib/IFG have no license information, but do mention specific authors
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@coleb can we get a license PR for enumerateheterocycles?

It's certainly good to be explicit, but I've just been assuming that things are covered by the license.txt that's in the top-level directory.
We should be a bit more careful with checking license in the future.

The two contrib changes are in: #1932 and #1933

Thanks!

@bp-kelley and @greglandrum , on vacation this week, but will definitely follow up when I'm back. So I know what is being asked, just asking that the copyright in rdkit/Chem/EnumerateHeterocycles.py be updated to reference the parent BSD license? or just removal of 'DE Shaw Research' entirely to make it easier going forward?
Would like to get it straightened out as I'd like to make a similar contribution to EnumerateHeterocycles for aliphatic systems by the UGM. :-)

@coleb : I think it would be fine to just add:
#  This file is part of the RDKit.
#  The contents are covered by the terms of the BSD license
#  which is included in the file license.txt, found at the root
#  of the RDKit source tree.

ping?

I've started the leg work on my side.

Any updates?

Just created pull request #1970 to include Greg's text after getting the internal clearance. Sorry for the delay!

Thanks!

Merged :)

Is that all of them so far?


Support for batch_size x num_heads x sequence_length x head_size

Hello,
In these lines (https://github.com/Dao-AILab/flash-attention/blob/187c2a06358d421e3d350fac8ff8714013c1f1cd/csrc/flash_attn/src/flash.h#L32-L40) it allows me to set the stride for accessing the various inputs. However, when I set the appropriate stride to access a tensor of shape batch_size x num_heads x sequence_length x head_size, I don't get expected results. In specific, I am hoping to use k and v with this BNSH format in the mha_fwd_kvcache api. Do you support or plan to support this input format?

You can just transpose the cache before passing in. There should be no extra copy.
# k_cache of shape (batch, nheads, seqlen, headdim)
k_cache = k_cache.transpose(1, 2)
# Pass k_cache to flash_attn_with_kvcache function


Update documentation with new ENV vars for postgres

Only update once postgres is working on dev site.

Finished today


username and passowrd

Type: Feature Request
i forggot my username
Azure Data Studio version: azuredatastudio 1.46.1 (ba29842b81dec01177415e53948ca2168e69c3f8, 2023-10-02T18:14:22.887Z)
OS version: Windows_NT x64 10.0.22000
Restricted Mode: No
Preview Features: Disabled
Modes:

Hi @jenna-lab
How can we help?

No enough info..


eggd_pandora initial development

eggd_pandora takes a case in OpenCGA (http://docs.opencb.org/display/iva) this app and submits the variants for the proband from the primary interpretation to the DECIPHER database (https://www.deciphergenomics.org/).
This app runs the script pandora.sh which runs pull_from_opencga.py, which is a python script that extracts the necessary information for each case to be submitted to DECIPHER and outputs it in a JSON called case_phenotype_and_variant_data.json, which is in the following format:
    case_dict = {
        'sex': sex,
        'clinical_reference': opencga_proband_id,
        'phenotype_list': phenotype_list,
        'variant_list': variant_list
    }

The pandora.sh script then inputs this JSON to the push_to_decipher.py script which reformats this information and submits it to DECIPHER.
An example of this process in this DNAnexus job Job-GP025VQ4fbKVk75K6FQyzFvX

This job took case  SAP-55748-1, as seen in OpenCGA here:

And uploaded it to DECIPHER including the variants and phenotypes, as shown below:

This change is

Example job with new features: job-GPFqkV04fbKy53QFY91QgG06

ClinVar development work has been moved and is now in this branch: https://github.com/eastgenomics/eggd_pandora/tree/clinvar


pt_BR translation improvements

Fixes Or Enhances
Make sure that you've checked the boxes below before you submit PR:

[x] Tests exist or have been written that cover this particular change.

@go-playground/validator-maintainers

Coverage remained the same at 74.177% when pulling 97dba8caaf703126389e9383f64aa79ddd51c5b6 on cyruzin:master into 9e2ea4038020b5c7e3802a21cfa4e3afcfdcd276 on go-playground:master.


Error calling Oracle RPCs

Take this call as an example:
fn collateral_to_wrapped(
    amount: BalanceWrapper<Balance>,
    currency_id: CurrencyId,
) -> Result<BalanceWrapper<Balance>, DispatchError>;

BalanceWrapper has the following concrete type:
pub struct BalanceWrapper {
    pub amount: u128,
}

For some reason polkadot-js/apps requires a second CurrencyId in the amount which results in the following error:

See your definitions (currencyId + amount) - https://github.com/interlay/interbtc-types/blob/bb448280cfba1ad585eb78880a94a5c0840a195b/definitions.json#L9-L12


New package: MoM_MPI v0.1.0

Registering package: MoM_MPI
Repository: https://github.com/deltaeecs/MoM_MPI.jl
Created by: @deltaeecs
Version: v0.1.0
Commit: 72a16b78eac4810af8b3ce927c0fb8efc0fff863
Reviewed by: @deltaeecs
Reference: https://github.com/deltaeecs/MoM_MPI.jl/commit/72a16b78eac4810af8b3ce927c0fb8efc0fff863#commitcomment-115334613
Description: MPI based parallel for method of moments (MoM) and MLFMA (https://github.com/deltaeecs/CEM_MoMs.jl)

abort


Update pricing.html

There is now an automated billing Option, so this section should be updated.

It might be good to also add info about the Discounts you get when paying with STORJ. I don't find the info now, though.

Tested locally, looks good, merging this in.


Stop Spinner from rotating when hidden.

Seems the currently used style does not stop the Rotator#PART_Rotator from rotating when IsIndeterminate="True". Found this using GPU resources in the background despite not being on screen. This fixes that by updating that current animation to override the Speed property for the entire duration of animation, forever.

This fix wont stops rotator control, it still calls OnSpeedChanged method. Good for try though.
I will do some improvements for it later.


Improvements to current website

Current changes:

Fix the inconsistent spacing on homepage, at representative text(compare with current website)(figma)
Fix border thickness to be more even on faq, before it was very thin when expanding a question(figma)
Change header to one line, to match figma(figma)
Change header size on company page, and faq, to be the same(figma)
Add product catalog button in company page(figma)
Change buttons in faq to have zoom in when hovered, and brighten up
Add transition to header on homepage
Add zoom in for signup button when hovering
Make carousel smaller in faq
Make the first letter in the first paragraph of the homepage to be larger(figma).

Font and size were changed to :

Header: text-5xl, font-medium
Subheader: text-4xl font-normal

Feel free to remove or edit any change, the transition in homepage does perhaps not look better.

I will create PR for prod, after it has been accepted


python3Packages.sourmash: 4.8.3 -> 4.8.4

Description of changes
Diff: https://github.com/sourmash-bio/sourmash/compare/v4.8.3...v4.8.4
Changelog: https://github.com/sourmash-bio/sourmash/releases/tag/v4.8.4
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[x] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[x] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 255999 run on x86_64-linux 1

4 packages built:

python310Packages.sourmash
python310Packages.sourmash.dist
python311Packages.sourmash
python311Packages.sourmash.dist


Should distinguish between build statuses

For versioning
Right now the versioning is based purely on the build ID. But the build ID is not a single, immutable snapshot of the external state, because a build can meaningfully evolve under the same ID number.
Instead, the version should be ID + a build status.
For example, 123+pending is different from 123+started, which is different again from 123+failed.
For filtering
Users of the resource will probably only be interested in one or two statuses. So it should be possible to filter -- "only show me new passes", etc, with "show me everything for every build" as the default behaviour. I'd like feedback on whether that's better as a get param or as a source param.

I will ignore filtering for now.


Ixia Vision xStream 40 (NPB) is showing up as Kentik Default - Can this be categorized as Networking Switch?

Ixia Vision xStream 40 is a network packet broker (NPB)
Currently discovering as Kentik Default.
Would like this categorized as Ixia Switch or under Switches.
SYSOBJECTID =  <IP_ADDRESS>.4.1.23<IP_ADDRESS>
That SysID shows the same on NR as it does SW.

Brad,
We will need an SNMP Walk from this device to build a profile for Ixia


Do polymorphic types registering from source still fail with recent Visual Studio compiler?

Hi!
As many others I'm adapting an existing serialization subsystem in my project from Boost -> Cereal. A big plus for Cereal is that I can actually read sources compared to Boost where I just give up :-)
My main question is in the issue title. What's wrong with Windows DLLs and/or Visual Studio that makes registering from sources impossible? Where is the source of the problem? It's not documented anywhere. Maybe it's already fixed in recent VS that has full C++14 support? Registering from sources is a HUGE advantage.
I was digging through the code and it seems that CEREAL_BIND_TO_ARCHIVES(...) is the trigger of pointer serialization code instantiation. If I place this macro in header (default registering from header approach) and then include this header in multiple cpp files, does it mean that serialization code will be instantiated multiple times in each translation unit?
I understand that static bindings map will store only one instance, and even "registering" should be called only once, because compiler will preserve only one instance of bind_to_archives<...> const & static variable. But what if I define serialization in one shared library and use it from another library (or executable) by including the header with CEREAL_BIND_TO_ARCHIVES(...)? Will it result in producing ptr serialization code in each library and multiple 'trials' to register it?

Seems that devs aren't so active here...
While waiting for the answer I did check by myself. Polymorphic types registering in cpp-file that is built into shared library (DLL), that in turn is utilized by executable test-case app result in NO errors. Serialization working fine. Will further test how will two DLLs interact, one exporting serialization API and another using it.
Visual Studio 2017 15.7.2.

@AzothAmmo thanks for the response!
BTW I've had to fix one minor issue for gcc/clang -- it's related to CEREAL_DLL_EXPORT expanding to nothing for these compilers. I build all modules in my project with -fvisibility=hidden which prevents all non-decorated symbols from being exported (like VS does by default). In that case you have to explicitly #define CEREAL_DLL_EXPORT __attribute__ ((visibility("default"))).

Update: I've implemented 2nd approach and it's actually working just fine.


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade-2.x" workflow

⚪ Backport skipped
The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.
Manual backport
To create the backport manually run:
backport --pr 537

Questions ?
Please refer to the Backport tool documentation


Add custom Github Action for running Go tests to terraform workflow file

Replace the existing workflow code with the new custom Github Action for running Go tests

Github still redacts the action secrets despite being passed in as inputs.

@areida says I am not authorized to merge this PR


Cant install editable package in conjunction with other packages

Issue description
When trying to install from my Pipfile (see below) it seems to be struggling to install the editable for the package.
If I remove the line snpit = {editable = true,path = "."} and run pipenv install --dev followed by pipenv install -e . it works fine. But trying to do it all in the one go with the editable snpit package seems to cause the error.
Expected result
Everything installs and I have access to my package inside the pipenv environment.
Actual result
$ pipenv install --verbose --dev
Creating a virtualenv for this project…
Pipfile: /home/michael/Downloads/tmp/snpit/Pipfile
Using /home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/bin/python3.7 (3.7.3) to create virtualenv…
⠴ Creating virtual environment...Already using interpreter /home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/bin/python3.7
Using real prefix '/home/linuxbrew/.linuxbrew/Cellar/python/3.7.3'
New python executable in /home/michael/.local/share/virtualenvs/snpit-qNOLdjLl/bin/python3.7
Also creating executable in /home/michael/.local/share/virtualenvs/snpit-qNOLdjLl/bin/python
Installing setuptools, pip, wheel...
done.

✔ Successfully created virtual environment!
Virtualenv location: /home/michael/.local/share/virtualenvs/snpit-qNOLdjLl
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
✔ Success!
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  black>=19.3b0
  coverage>=4.5.3
  pytest>=5.0

Finding the best candidates:
  found candidate black==19.3b0 (constraint was >=19.3b0)
  found candidate coverage==5.0a6 (constraint was >=4.5.3)
  found candidate pytest==5.0.1 (constraint was >=5.0)

Finding secondary dependencies:
  coverage==5.0a6 not in cache, need to check index
  coverage==5.0a6           requires -
  pytest==5.0.1 not in cache, need to check index
  pytest==5.0.1             requires atomicwrites>=1.0, attrs>=17.4.0, importlib-metadata>=0.12, more-itertools>=4.0.0, packaging, pluggy<1.0,>=0.12, py>=1.5.0, wcwidth
  black==19.3b0 not in cache, need to check index
  black==19.3b0             requires appdirs, attrs>=18.1.0, click>=6.5, toml>=0.9.4

New dependencies found in this round:
  adding ['appdirs', '', '[]']
  adding ['atomicwrites', '>=1.0', '[]']
  adding ['attrs', '>=17.4.0,>=18.1.0', '[]']
  adding ['click', '>=6.5', '[]']
  adding ['importlib-metadata', '>=0.12', '[]']
  adding ['more-itertools', '>=4.0.0', '[]']
  adding ['packaging', '', '[]']
  adding ['pluggy', '<1.0,>=0.12', '[]']
  adding ['py', '>=1.5.0', '[]']
  adding ['toml', '>=0.9.4', '[]']
  adding ['wcwidth', '', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  appdirs
  atomicwrites>=1.0
  attrs>=17.4.0,>=18.1.0
  black>=19.3b0
  click>=6.5
  coverage>=4.5.3
  importlib-metadata>=0.12
  more-itertools>=4.0.0
  packaging
  pluggy<1.0,>=0.12
  py>=1.5.0
  pytest>=5.0
  toml>=0.9.4
  wcwidth

Finding the best candidates:
  found candidate appdirs==1.4.3 (constraint was <any>)
  found candidate atomicwrites==1.3.0 (constraint was >=1.0)
  found candidate attrs==19.1.0 (constraint was >=17.4.0,>=18.1.0)
  found candidate black==19.3b0 (constraint was >=19.3b0)
  found candidate click==7.0 (constraint was >=6.5)
  found candidate coverage==5.0a6 (constraint was >=4.5.3)
  found candidate importlib-metadata==0.19 (constraint was >=0.12)
  found candidate more-itertools==7.2.0 (constraint was >=4.0.0)
  found candidate packaging==19.1 (constraint was <any>)
  found candidate pluggy==0.12.0 (constraint was >=0.12,<1.0)
  found candidate py==1.8.0 (constraint was >=1.5.0)
  found candidate pytest==5.0.1 (constraint was >=5.0)
  found candidate toml==0.10.0 (constraint was >=0.9.4)
  found candidate wcwidth==0.1.7 (constraint was <any>)

Finding secondary dependencies:
  coverage==5.0a6           requires -
  wcwidth==0.1.7 not in cache, need to check index
  wcwidth==0.1.7            requires -
  more-itertools==7.2.0 not in cache, need to check index
  more-itertools==7.2.0     requires -
  pluggy==0.12.0 not in cache, need to check index
  pluggy==0.12.0            requires importlib-metadata>=0.12
  importlib-metadata==0.19 not in cache, need to check index
  importlib-metadata==0.19  requires zipp>=0.5
  py==1.8.0 not in cache, need to check index
  py==1.8.0                 requires -
  click==7.0 not in cache, need to check index
  click==7.0                requires -
  atomicwrites==1.3.0 not in cache, need to check index
  atomicwrites==1.3.0       requires -
  black==19.3b0             requires appdirs, attrs>=18.1.0, click>=6.5, toml>=0.9.4
  pytest==5.0.1             requires atomicwrites>=1.0, attrs>=17.4.0, importlib-metadata>=0.12, more-itertools>=4.0.0, packaging, pluggy<1.0,>=0.12, py>=1.5.0, wcwidth
  packaging==19.1 not in cache, need to check index
  packaging==19.1           requires attrs, pyparsing>=2.0.2, six
  attrs==19.1.0 not in cache, need to check index
  attrs==19.1.0             requires -
  appdirs==1.4.3 not in cache, need to check index
  appdirs==1.4.3            requires -
  toml==0.10.0 not in cache, need to check index
  toml==0.10.0              requires -

New dependencies found in this round:
  adding ['pyparsing', '>=2.0.2', '[]']
  adding ['six', '', '[]']
  adding ['zipp', '>=0.5', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3
Current constraints:
  appdirs
  atomicwrites>=1.0
  attrs>=17.4.0,>=18.1.0
  black>=19.3b0
  click>=6.5
  coverage>=4.5.3
  importlib-metadata>=0.12
  more-itertools>=4.0.0
  packaging
  pluggy<1.0,>=0.12
  py>=1.5.0
  pyparsing>=2.0.2
  pytest>=5.0
  six
  toml>=0.9.4
  wcwidth
  zipp>=0.5

Finding the best candidates:
  found candidate appdirs==1.4.3 (constraint was <any>)
  found candidate atomicwrites==1.3.0 (constraint was >=1.0)
  found candidate attrs==19.1.0 (constraint was >=17.4.0,>=18.1.0)
  found candidate black==19.3b0 (constraint was >=19.3b0)
  found candidate click==7.0 (constraint was >=6.5)
  found candidate coverage==5.0a6 (constraint was >=4.5.3)
  found candidate importlib-metadata==0.19 (constraint was >=0.12)
  found candidate more-itertools==7.2.0 (constraint was >=4.0.0)
  found candidate packaging==19.1 (constraint was <any>)
  found candidate pluggy==0.12.0 (constraint was >=0.12,<1.0)
  found candidate py==1.8.0 (constraint was >=1.5.0)
  found candidate pyparsing==2.4.2 (constraint was >=2.0.2)
  found candidate pytest==5.0.1 (constraint was >=5.0)
  found candidate six==1.12.0 (constraint was <any>)
  found candidate toml==0.10.0 (constraint was >=0.9.4)
  found candidate wcwidth==0.1.7 (constraint was <any>)
  found candidate zipp==0.5.2 (constraint was >=0.5)

Finding secondary dependencies:
  six==1.12.0 not in cache, need to check index
  six==1.12.0               requires -
  more-itertools==7.2.0     requires -
  toml==0.10.0              requires -
  attrs==19.1.0             requires -
  atomicwrites==1.3.0       requires -
  click==7.0                requires -
  black==19.3b0             requires appdirs, attrs>=18.1.0, click>=6.5, toml>=0.9.4
  coverage==5.0a6           requires -
  importlib-metadata==0.19  requires zipp>=0.5
  pyparsing==2.4.2 not in cache, need to check index
  pyparsing==2.4.2          requires -
  py==1.8.0                 requires -
  wcwidth==0.1.7            requires -
  packaging==19.1           requires attrs, pyparsing>=2.0.2, six
  pytest==5.0.1             requires atomicwrites>=1.0, attrs>=17.4.0, importlib-metadata>=0.12, more-itertools>=4.0.0, packaging, pluggy<1.0,>=0.12, py>=1.5.0, wcwidth
  appdirs==1.4.3            requires -
  zipp==0.5.2 not in cache, need to check index
  zipp==0.5.2               requires -
  pluggy==0.12.0            requires importlib-metadata>=0.12
------------------------------------------------------------
Result of round 3: stable, done

Locking [packages] dependencies…
✘ Locking Failed!
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  file:///home/michael/Downloads/tmp/snpit (from -r /tmp/pipenvvtjf1v9vrequirements/pipenv-c7xg5up_-constraints.txt (line 3))
  pysam==0.15.2

Finding the best candidates:
  found candidate -e file:///home/michael/Downloads/tmp/snpit (constraint was <any>)
  found candidate pysam==0.15.2 (constraint was ==0.15.2)

Finding secondary dependencies:
  pysam==0.15.2             requires -
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 119, in main
    parsed.requirements_dir, parsed.packages)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 726, in resolve_deps
    req_dir=req_dir,
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 385, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 206, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 280, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 234, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 426, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(download_dir, ireq, wheel_cache)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 297, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 260, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 197, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 321, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 127, in prep_for_dist
    self.req.run_egg_info()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 474, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /home/michael/Downloads/tmp/snpit/
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple

                          ROUND 1
Current constraints:
  file:///home/michael/Downloads/tmp/snpit (from -r /tmp/pipenvvtjf1v9vrequirements/pipenv-c7xg5up_-constraints.txt (line 3))
  pysam==0.15.2

Finding the best candidates:
  found candidate -e file:///home/michael/Downloads/tmp/snpit (constraint was <any>)
  found candidate pysam==0.15.2 (constraint was ==0.15.2)

Finding secondary dependencies:
  pysam==0.15.2             requires -
Traceback (most recent call last):
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 119, in main
    parsed.requirements_dir, parsed.packages)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 85, in _main
    requirements_dir=requirements_dir,
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/resolver.py", line 69, in resolve
    req_dir=requirements_dir
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 726, in resolve_deps
    req_dir=req_dir,
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 385, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 206, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/resolver.py", line 280, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 234, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 426, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(download_dir, ireq, wheel_cache)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 297, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 260, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/resolve.py", line 197, in _get_abstract_dist_for
    req, self.require_hashes, self.use_user_site, self.finder,
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 321, in prepare_editable_requirement
    abstract_dist.prep_for_dist(finder, self.build_isolation)
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 127, in prep_for_dist
    self.req.run_egg_info()
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/req/req_install.py", line 474, in run_egg_info
    command_desc='python setup.py egg_info')
  File "/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/patched/notpip/_internal/utils/misc.py", line 705, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in /home/michael/Downloads/tmp/snpit/

Steps to replicate
I am trying to install this package locally.
git clone https://github.com/mbhall88/snpit.git
cd snpit
make install
# runs `pipenv install --dev` plus some cleaning

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.
$ pipenv --support
Pipenv version: '2018.11.26'
Pipenv location: '/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv'
Python location: '/home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/bin/python3.7'
Python installations found:

3.7.3: /home/linuxbrew/.linuxbrew/bin/python3.7
3.7.3: /home/linuxbrew/.linuxbrew/bin/python3.7m
3.6.8: /usr/bin/python3.6
3.6.8: /usr/bin/python3.6m
2.7.15: /usr/bin/python2

PEP 508 Information:
{'implementation_name': 'cpython',
 'implementation_version': '3.7.3',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-1045-oem',
 'platform_system': 'Linux',
 'platform_version': '#50-Ubuntu SMP Wed Jun 26 11:16:36 UTC 2019',
 'python_full_version': '3.7.3',
 'python_version': '3.7',
 'sys_platform': 'linux'}

System environment variables:

PATH
GREP_COLOR
CLUTTER_IM_MODULE
LS_COLORS
LC_MEASUREMENT
LC_PAPER
LC_MONETARY
XDG_MENU_PREFIX
_
LANG
LESS
DISPLAY
OLDPWD
GNOME_SHELL_SESSION_MODE
EDITOR
COLORTERM
USERNAME
CHROME_DESKTOP
NO_AT_BRIDGE
XDG_VTNR
GIO_LAUNCHED_DESKTOP_FILE_PID
ZSH
SSH_AUTH_SOCK
MANDATORY_PATH
LC_NAME
XDG_SESSION_ID
USER
PAGER
LSCOLORS
DESKTOP_SESSION
QT4_IM_MODULE
TEXTDOMAINDIR
DEFAULTS_PATH
PWD
HOME
LC_CTYPE
TEXTDOMAIN
SSH_AGENT_PID
TERM_PROGRAM
TERM_PROGRAM_VERSION
QT_ACCESSIBILITY
XDG_SESSION_TYPE
XDG_DATA_DIRS
XDG_SESSION_DESKTOP
LC_ADDRESS
GJS_DEBUG_OUTPUT
LC_NUMERIC
HOMEBREW_REPOSITORY
GTK_MODULES
PAPERSIZE
VISUAL
WINDOWPATH
TERM
SHELL
QT_IM_MODULE
XMODIFIERS
IM_CONFIG_PHASE
XDG_CURRENT_DESKTOP
GPG_AGENT_INFO
GIO_LAUNCHED_DESKTOP_FILE
SHLVL
XDG_SEAT
HOMEBREW_PREFIX
MANPATH
LC_TELEPHONE
GDMSESSION
GNOME_DESKTOP_SESSION_ID
LOGNAME
DBUS_SESSION_BUS_ADDRESS
XDG_RUNTIME_DIR
XAUTHORITY
XDG_CONFIG_DIRS
LC_IDENTIFICATION
HOMEBREW_CELLAR
INFOPATH
GJS_DEBUG_TOPICS
DESKTOP_MODE
SESSION_MANAGER
GTK_IM_MODULE
LC_TIME
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PIP_SHIMS_BASE_MODULE
PIP_PYTHON_PATH
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:
Debug–specific environment variables:

PATH: /home/linuxbrew/.linuxbrew/Cellar/pipenv/2018.11.26_2/libexec/tools:/home/michael/.cargo/bin:/home/michael/.cargo/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL: /home/linuxbrew/.linuxbrew/bin/zsh
EDITOR: vim
LANG: en_GB.UTF-8
PWD: /home/michael/Downloads/tmp/snpit

Contents of Pipfile ('/home/michael/Downloads/tmp/snpit/Pipfile'):
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
coverage = ">=4.5.3"
black = ">=19.3b0"
pytest = ">=5.0"

[packages]
pysam = "==0.15.2"
snpit = {editable = true,path = "."}

[pipenv]
allow_prereleases = true

哦

I'm having this same issue
pipenv install --dev fails, if I remove my local dependency, it works fine, then I install the local dependency afterwards:
pipenv install --dev # fails

when I change my Pipfile it works fine:
From this
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
pygqlc = "*"
inflect = "*"
pydash = "*"
click = "*"
valiotworker = "*"
pytz = "*"
tzlocal = "*"
attrs = ">=19.0.0"

[dev-packages]
pytest = "*"
pytest-cov = "*"
black = "*"
twine = "*"
pylint = "*"
gstorm = {editable = true,path = "."}

[requires]
python_version = "3.7"

[pipenv]
allow_prereleases = true

to this:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[packages]
pygqlc = "*"
inflect = "*"
pydash = "*"
click = "*"
valiotworker = "*"
pytz = "*"
tzlocal = "*"
attrs = ">=19.0.0"

[dev-packages]
pytest = "*"
pytest-cov = "*"
black = "*"
twine = "*"
pylint = "*"

[requires]
python_version = "3.7"

[pipenv]
allow_prereleases = true

Thanks for the report and sorry for the trouble -- can you confirm whether this is an issue if you install the latest pre-release of pipenv via pip install --upgrade --pre pipenv?

Sorry @techalchemy but I will have to let you debug this with @barucAlmaguer as I no longer use pipenv and the project I raised the issue with no longer uses it. Feel free to close the issue if you don't get any responses.

I believe this is resolvable now -- but let me know if its still an issue on pipenv==2022.8.19


What does bbox_mAP_s means?

Including [bbox_mAP_s, bbox_mAP_m, bbox_mAP_l] when running a fasterRCNN network.
It seems not commonly seen elsewhere, and I couldn't google anything related to them.
I guess if they are three scale of boxes?

These are metrics for small, medium and large objects, refer to https://cocodataset.org/#detection-eval for detail.


Field mapping, multiple html source regex capture

When there are up to 4 html source and regex capture expressions, it seems that the 5th ore bigger than can't be added to the list of fields to be matched, trough  regex expression.
Using web admin interface, chrome.
It gives errors on save/update/add new HtmlSource to field via regex capture expression, on web admin interface, i'v teste more than 100 times on different scenarios. It's imposible to add more htmlsource extracrors via regex.
Is this en error or there's a maximum number of variables to field via regex capture?

Or it is recommended to edit the parsers.xml for editing parsers?

Plus, tested the scenario where having multiple HTML Parsers in one index, to work on different htmlsource (html) regex capture blocks, but still not working and even crashing the data gathering process.


feat: update group permissions

What this PR does 📖

Renamed SetGroupName to EditGroupInfo

Added RemoveParticipants permission. If someone has this they can remove a user from a group

EditGroupInfo now allows: Renaming group name + description, changing group icon and banner

Changed no permission error from PublicKeyInvalid to Unauthorized

Updates the tests which only used direct conversation instead of group

Left some comments. Not sure I like or completely agree with moving multiple functionalities under a single permission here (eg changing/updating name, images, etc).

was talking to hugo about that. and we decided for name, description, icon and banner we would use one permission

was talking to hugo about that. and we decided for name, description, icon and banner we would use one permission

Not sure if it should be that narrow imho

Can you correct the test?


Support custom colours

Would it be possible to add a means to define new colours, for use with terminals that support a broader range of them?

Alternately, would you accept a PR to add the "light" colour variants for 16 colour terminals?

The light one should already be supported via dim.


Model registry proposal

Proposal for Model Registry for Open Data Hub
Description
Adding written proposal for Model Registry.
How Has This Been Tested?
TODO
Merge criteria:

[x] The commits are squashed in a cohesive manner and have meaningful messages.
[ ] Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
[ ] The developer has manually tested the changes and verified that the changes work

Squashed commits.

@rimolive have you had a chance to review?
@taneem-ibrahim, @anishasthana can this PR be merged? We are working on the implementation details and will have those details to add in a few days.

Given the direction we are taking, is this still relevant to keep here? @dhirajsb @danielezonca

@rareddy, it would be useful to share the proposal contents with the kubeflow community to share some design and implementation goals.

@dhirajsb I submitted a PR to your GH with changes to the above PR with modified arch stuff after the shift in direction. I do not see that here.

/lgtm

This PR has been approved by the stakeholders. Merging.


Refactor/crud to resource

refactors index, show, create, update and destroy methods into resource.
still need to create a singleton resource subclass to handle things like org and user.

Coverage remained the same at 100.0% when pulling 4fb92fb66a0f3252164849f24d5e016794c08852 on refactor/crud-to-resource into fd77a67009d85e30e4c64e806353784b636d9b4d on master.

Coverage remained the same at 100.0% when pulling 38648a9e4ce8547e8da155bdf7634d995272fc3a on refactor/crud-to-resource into fd77a67009d85e30e4c64e806353784b636d9b4d on master.

Coverage remained the same at 100.0% when pulling c5b5bef5981d3f1bc22af509183050277229c4dc on refactor/crud-to-resource into fd77a67009d85e30e4c64e806353784b636d9b4d on master.


Not recognizing US holidays

The latest update is not recognizing US based holidays. This past Monday was Labor Day. Trash pickup is usually Thursdays. It did not advance a day. Screen shots below.

I'm seeing this issue as well.  BTW, I just installed this integration this past weekend and it was factoring in the labor day holiday when I first installed it.  I think I updated it at least once if not more since then so maybe one of the updates changed something?

Can you guys check the 3.06 version? It should fix it - hopefully before you pull your trash out a day earlier :).

@patdemko , the issue was that the integration was loading the holdays from today onwards. So on the weekend, it calculated properly, but from Tuesday it stopped seeing the holiday. Thanks for the catch @PrayerfulDrop

@bruxy70 that fixed it!  Great job!!

@patdemko , the issue was that the integration was loading the holdays from today onwards. So on the weekend, it calculated properly, but from Tuesday it stopped seeing the holiday. Thanks for the catch @PrayerfulDrop

Yeah, that fixed my issue as well.  Thanks for the very fast fix!


Esp32 AT Commands Azure IoT Hub.

Hi i have a question regarding Azure Iot Hub connection.
Following : https://blog.csdn.net/espressif/article/details/105045610,  i connected to the hub.
Using the SAS token created from symmetric key.
But without sending any CA certificate!!! to myIoThub.azure-devices.net.
I also enstablished a single TLS connection to the hub also without the certificate.
In Microsoft Documentation TLS/SSL configuration :
To use the MQTT protocol directly, clients must connect via TLS/SSL. Attempts to skip this step fail with a connection error.
To establish a TLS connection, you may need to download and reference the DigiCert Baltimore root certificate. This certificate is used by Azure to secure the connection.
In all examples Python, C SDK for Azure, Arduino, the first thing to do is to give the CA certificate.
Why ESP32 AT doesn't need to verify any CA certificate?
Thanks.

@sl2121 For a common TLS connection, certificate should be an optional item, not a required item. More details are in RFC 5246, search Message flow for a full handshake.
so as for ESP32-AT, MQTT can be established without certificate verify, that should make sense.
Of course, you can specify the certificate verification mode by <scheme> parameter, which defined in AT+MQTTUSERCFG command.
Please let me know if anything unclear.


Fix next-bechdel README's formatting and clarity

This PR attempts to fix next-bechdel README's formatting and clarify the header definitions. The header tables in the README are now formatted correctly.

@dmil when you have time, could you please have a look at my changes? thank you


Proxy release 2024-04-04

Proxy release 2024-04-04
Please merge this Pull Request using 'Create a merge commit' button

I would wait for: https://github.com/neondatabase/aws/pull/1225
And + 20 minutes to check, that consumption events are pushed to s3 correctly.


enable zRAM

@shaunmulligan is this still required? Where did the requirement come from?

@willnewton its not required, but would be nice to know if it would give us any visible improvements. At the time I had read that some of the OSes for low spec machines like ChromeOS had enabled it, so thought it would be worth looking into. Do you have any experience with it?

@willnewton I need zRam for a project which will either be based on resinOS or at least closely resembling. Should I do a PR when I have it working?

@landswellsong can you give us a bit more context on this, how it is contributing to your requirements? This would help us prioritize things, and make sure everything fits well together. Thanks!

@imrehg we run multiple containers on one [possibly memory starved] device. It [usually] doesn't have a swap because everything is either SD or eMMC and the containers that are run are community-provided thus not guaranteed to be careful with memory allocations. Having zRam would give a huge asset for dealing with stale memory.

@landswellsong I opened a PR for adding the appropriate kernel modules here: https://github.com/resin-os/meta-resin/pull/851
I'm not sure there is a way we can have it running generically as the size of the disk is quite hardware and workload specific.

zram as per #851 is currently merged but the module is not enabled by default, nor is any zram swap added.
Although most devices have emmc storage it might be also worth adding support for a small swap file. This can be especially useful with debug builds of applications that require additional resources.

@splitice Currently the idea is that resinOS brings in the kernel module and offers the possibility for the user to set zram swap from the application/container. Isn't that covering your use-case? If not, could you detail?

@agherzan sorry I was not aware that the permissions were relaxed that far. I will certainly try that.

I will close this ticket for now. Do come back if you need any support @splitice .


Enable remarks for ttgir lowering with SourceMgrDiagnosticHandler

This will print remarks like below
01-vector-add.py:48:16: remark: vec = 4
x = tl.load(x_ptr + offsets, mask=mask)
^
With printOpOnDiagnostic being true, it will show
01-vector-add.py:49:16: remark: vec = 4
y = tl.load(y_ptr + offsets, mask=mask)
^
01-vector-add.py:49:16: note: see current operation: %332 = "tt.load"(%315, %165) <{cache = 1 : i32, evict = 1 : i32, isVolatile = false, operandSegmentSizes = array<i32: 1, 1, 0>}> : (tensor<1024x!tt.ptr<f32, 1>, #triton_gpu.blocked<{sizePerThread = [4], threadsPerWarp = [32], warpsPerCTA = [4], order = [0]}>>, tensor<1024xi1, #triton_gpu.blocked<{sizePerThread = [4], threadsPerWarp = [32], warpsPerCTA = [4], order = [0]}>>) -> tensor<1024xf32, #triton_gpu.blocked<{sizePerThread = [4], threadsPerWarp = [32], warpsPerCTA = [4], order = [0]}>>

I am thinking about using remarks to capture key optimization information and the information can be integrated into a performance analysis tool (Proton may be a good choice). CC @Jokeren
I am interested in the following information, not sure which info will be from the compiler and which can be from Proton:

number of layout conversions, memory transfer size for each conversion
number of loads/stores, for each load, store, is it coalesced? Is it vectorized?
For each loop, is SWP efficiently enabled?
Which mma instruction is used for tl.dot? Will transposing make tl.dot eligible for mma v3?

Remarks can show location for both .py source and .ttgir file.
I haven't figured out how to filter Diagnostics for SourceMgrDiagnosticHandler. Currently we will also get some warnings:
01-vector-add.py:28:0: warning: Unhandled parameter attribute 'tt.divisibility'
If printOpOnDiagnostic is on, it will dump the whole function.

This is interesting. I need to think more about it though.
Could we keep it as a draft PR and come back discussing it later. I promise that I won't forget :)

Saved it to my github folder.

Hi @manman-ren , I had a discussion with @htyu and understand the purpose of the PR. I'm very supportive on providing more metadata.
I have two suggestions.

We could provide the kernel source file path to proton and let it analyzes the triton GPU IR using existing passes to recover information.
Once we have all the information, we could first reuse our existing viewer to view at most two metrics on the terminal. Next, maybe it's time to start integrating it with some existing visualizers because we need to view multiple metrics and source lines.

I may spend some time working on this soon :) Happy to chat if you have any thoughts

Hi @manman-ren , I had a discussion with @htyu and understand the purpose of the PR. I'm very supportive on providing more metadata.
I have two suggestions.

We could provide the kernel source file path to proton and let it analyzes the triton GPU IR using existing passes to recover information.
Once we have all the information, we could first reuse our existing viewer to view at most two metrics on the terminal. Next, maybe it's time to start integrating it with some existing visualizers because we need to view multiple metrics and source lines.

I may spend some time working on this soon :) Happy to chat if you have any thoughts

Thanks for taking the time!
For 1> My current plan is to emit optimization remarks (it will come with source line and ttgir line). Maybe Proton can parse the remarks. I am not sure how easy it is to let Proton run passes on GPU IR. I have a follow up patch that tries to add a pass to collect information and to emit optimization remarks from lowering ttgir to llvm.
For 2> It will be great if we can have a visualizer to look at profiling data together with analysis data, we can have one tool for AMD/NV and also for PT2/Triton.

For 1> My current plan is to emit optimization remarks (it will come with source line and ttgir line). Maybe Proton can parse the remarks. I am not sure how easy it is to let Proton run passes on GPU IR. I have a follow up patch that tries to add a pass to collect information and to emit optimization remarks from lowering ttgir to llvm.

Using a separate pass to get the information is better than instrumenting existing compiler passes.
But still, I'm worried about parsing the remark.
If instead we parse the GPU IR by calling module = ir.parse_mlir_module(full_name, context), and use an analysis function to analyze the IR, it will offer the following benefits:

This function can utilize all existing analysis modules.
Proton can get the parsed information in memory without parsing the file.
The analysis function can be called as a python interface of triton and benefit other profilers or analyzers.

The design would be
class result:
    file: str
    line_no: int
    column_no: int
    remarks: dict[str, problem]

class func_result:
   results: list[result]

class module_result:
    func_results: dict[str, func_result]

module = ir.parse_mlir_module(<gpu ir>)
analysis_results = libtriton.analyze_module(module)
libproton.inspect(analysis_results)

Yeah, I added a separate pass but there are decisions made during the lowering phase, such as vectorization, if a load instruction will be coalesced etc. For those, I am hoping to use optimization remarks.
https://github.com/openai/triton/pull/3853 is adding a perf-collection pass. It is in draft mode, mostly for discussion purpose.

Discussions are now in the new PR: https://github.com/triton-lang/triton/pull/3922
closing this


memory leak using $http.get()

after using $http.get() function heap and listeners grow. Heap may be collected a bit, but not completely, listeners are never collected.   http://plnkr.co/oDIvkY

I am very happy that you could write a reproduce scenario and I will be extra happy when a fix will be available ;-)

@angufix do you have any idea about why that happens?

Now I'm a bit unsure whether it is a real memory leak, because if you take 2 snapshots and compare them there are only system things (surrounded with round brackets) left allocated. Timeline and Heaprecorder still show some leaks.
So the question is which tool should I trust?
I looked around in angular.js especially in createHttpBackend function. an xhr object is created first
var xhr = createXhr();

then it is filled with extra stuff like  callbacks for onload, onerror methods. In the completeRequest function xhr was set to null, but those callbacks were still unset, so a possible solution could be :
function completeRequest(callback, status, response, headersString, statusText) {
      // cancel timeout and subsequent timeout promise resolution
      if (timeoutId !== undefined) {
        $browserDefer.cancel(timeoutId);
      }
      xhr.onload = null;
      xhr.onerror = null;
      xhr.onabort = null;
      jsonpDone = xhr = null;

      callback(status, response, headersString, statusText);
      $browser.$$completeOutstandingRequest(noop);
    }

in the time line we have then only one listener pro http request. What do you think about it?

@angufix Can you create a custom build of angular with that change and check if it improves the situation?

@Narretz  I'm not angular developer and currently I cann't spent much time on setupping angular build environment. What I've done is to hack the 1.4.7  version of angular.js. as I described, and that was the solution to me.

@angufix @Narretz I will put the hack in our solution and test our memory problem and let you know the result.

@angufix Just press the 'Collect garbage' button in the middle of profiling, and you'll see that all the listeners are GC'ed. There is no leak.

I think there are still an issue.

To get this result, I click during 90 secondes on button "test mem leak" and I force GC at the 92th second.
I think there are two issues:

The number of listeners grow up without any reason. It seems that the "automatic" GC and the "forced" GC work differently because when "automatic" GC is executed only JS Heap is reduce and not the number listeners. It is necessary to force GC to "reset" correctly the number of listener.
When I "forced" GC the JS Heap do not retrieve the initial size (at startup JS Heap is equal to 9 300 000 and after "forced" GC JS Heap is equal to 9 560 000). So maybe there is a leak.

Maybe it is a normal behavior but I need to understand why.
Thank you for your help.

@tchavanis I understand that you have a need to explain this, but at the end of the day, that discussion needs to happen within the Chrome project (or within the specific browser that you would like to explain this behavior at) and not in Angular.


Error Occurs when I add new NECKS and so on.

Firstly, I train the model.
Then I need to modify the code again.
Specifically, I add new NECKS or HEADS.
During my debugging, the newly added NECKS reports 'compile error'.
Then I find that the running model will report the error due to 'compile error' caused by my modification.
I think the running code before my modification is loaded in CPU Memory, but why does the running program report the same error?
So how can solve this case?

Thanks for your help. @ZCMax

Sorry for the late reply, compile error should not appear after only adding necks and heads, pleas post your error log, and what code have you added in details.

Close due to inactivity.


Cosmos DB | Update for Az.CosmosDB 0.1.4 | MongoDB

Description
Update samples to reflect changes in Az.CosmosDB 0.1.4 release. This PR is for Mongo DB-specific samples.
Checklist

[X] This pull request was tested on both of:

[X] PowerShell 5.1 (Windows)
[ ] PowerShell 6.x
[X] PowerShell 7.x (Latest PowerShell)

[X] Scripts do not contain static passwords or other secret tokens.
[X] All Azure resource identifiers which must be universally unique are guaranteed to be so.

Testing information
Windows 10.0.18363, Powershell 5.1, Az 3.8.0
Windows 10.0.18363, Powershell 7.0.0, Az 3.8.0

#sign-off


220GB RAM cannot run 20%  waymo dataset.

Hi,
Devices with 220GB RAM and 4*3090 cannot run CenterPoint even though each GPU has only one batch size of waymo dataset.
My devices and environment  is as follows:
4 RTX 3090
220 GB RAM
Xeon E5 12
Ubuntu 18.04
torch 1.9
waymo-open-dataset-tf-2-0-0

When I run any model, the waymo dataloader will gradually fill up the memory (220GB) before running the training code,  as follows:

However,  when we only train 5% of the dataset, the code can run.

whether issue #1130 can help?


@orta => [Artwork] Enable Markdown support in "More Info: view

Fixes #736.

What’s the problem here with the github credentials being required on CircleCI?

No idea, we've not changed anything to my knowledge ( unless we've somehow revoked the API keys )

Oh, I now remember @ashfurrow saying something about revoking keys because CircleCI said we were leaking them, don’t know specifics yet, though.

Fixed the CI issues.
@jorystiefel Can you rebase this branch please?

the build timeout'd so I'm re-running, if not I think this is good to go.

:+1:


mv file.rs new_file.rs does not work

Describe The Bug
According to docs
mv ./file.rs ./new_file.rs

should rename file.
Instead of this it creates new new_file.rs directory and moves file.rs to new_file.rs/file.rs.

i see I'm doing
fs::copy(source_path_str, target_path_str) which i figured would work correctly.
meaning dir path would be parent, file path would be actual target.
anyhow, would you like to PR it?

In first view the issue should be in this line:
https://github.com/sagiegurari/duckscript/blob/43c175f3d66a04449ed92b43f465a9cdbbe66548/duckscript_sdk/src/sdk/std/fs/mv/mod.rs#L45
The doc says that is_file return true only if file exists.

ah right.... should have checked if ends with a separator / or \ instead

https://github.com/sagiegurari/duckscript/pull/320

@burrbull i've merged your PR (sorry for long wait) and will be releasing a new duckscript version in the next hours. thanks a lot for the help.


[Docs] MovR Tutorial for jOOQ

Background: https://airtable.com/tblD3oZPLJgGhCmch/viw1DKmbKhg2MIECH/rec0z6e6fCvHxdzDo
Description: CockroachDB now provides application code and documentation for using MovR with jOOQ
Each ORM has a simple getting started guide. This is great to unblock basic usage by doesn't provide extensive guidance to application developers for how to use CockroachDB with a robust application.
Team: Andrew Woods, Rafi Shamim, Eric Harmeling
Github Tracking Issue:

This isn't happening for 20.1.
Note: We have a a separate issue for the general "Build an App" tutorial for this ORM.


Update KMS client library

This Tink extension currently depends on the deprecated google.golang.org/api/cloudkms/v1 package for its KMS client library.
This PR updates the KMS client library to use the recommended cloud.google.com/go/kms/apiv1 package instead.

Hi @chuckx, thanks for taking a look at this.
I wanted to say that I'm happy to do further work on this PR if it's necessary. I'm not familiar with Bazel or Copybara, and I'm reasonably new to Go, so any feedback no-matter how minor is welcome. Thanks again.

Thanks for the PR. I'll be taking a closer look at this later this week.

Did you have any initial thoughts on this @chuckx?


Improve file path reading in the client tool.

This PR has the following changes:

Move Input/output dir from the config.yaml file to args so that the behavior is not affected by where the user runs the binary.  Default path: client/input and client/output.
Add a validation step with instruction messages if user does't run the tool from the recommended directory (i.e., repo root).
Add an optional arg that allows user to specify the path to the config file.
I found a bug in the macro substitution about file name matching for sub-dirs in the input/output and fixed it.
Improve the progress logs in the console to make it more readable to users.

wfm

Here are my proposed changes to this branch:
https://github.com/xiashang0624/dwh-migration-tools/compare/client-path-fix...ajwelch4:client-path-fix
First, we should instruct the user to use a virtualenv to prevent dependency conflicts between our tool and the tools installed at the system-level by the system package manager. This goes without saying to Python folks as it's just good hygiene but not all of our customers will know this. It's particularly important in our case because we are asking our users to install gcloud and this is usually installed at the system-level.  gcloud is written in Python and likely uses many of the same Google Cloud Python libs that we are and we can't count on users having the same version of gcloud and its transitive dependencies installed as we do. Most folks use Pipenv or Poetry nowadays to manage virtualenvs but the simplest way to convey the recommendation is to show the old fashioned way:
python3 -m venv venv
source venv/bin/activate

Pip, Pipenv and Poetry would also take care of downloading our tool if we were publishing it to PyPI but we are not so git clone it is:
git clone git@github.com:google/dwh-migration-tools.git

At this point, our PWD looks like this:
tree -L 1

.
├── dwh-migration-tools
└── venv

If the user doesn't plan on editing the tool and just wants to install it and get going:
pip3 install ./dwh-migration-tools/client

The above will invoke pips dependency resolver to install a set of concrete dependencies from the set of abstract dependencies listed in setup.py.:
https://github.com/ajwelch4/dwh-migration-tools/blob/client-path-fix/client/setup.py#L23-L33
Using abstract dependencies in setup.py will allow us to support library use cases and PyPI distribution someday if we so chose. I would recommend tests before chosing so ;-). PSO is already investigating how to string together this tool, DTS, DVT, etc. so the library use case may be useful.
If the user wants to ensure they get the same exact concrete dependencies as us, they can run:
pip3 install -r ./dwh-migration-tools/client/requirements.txt
pip3 install ./dwh-migration-tools/client

Re: populating requirements.txt. The Python world has largely moved to lock files for concrete dependency management and Pipenv and Poetry would generate a lock file for us from the abstract dependencies. Instead, we will generate requirements.text from the abstract dependencies in setup.py by running:
make requirements

If the user wants to edit the tool, they can install it in editable mode using pip's -e option:
pip3 install -e ./dwh-migration-tools/client

Another benefit of using setup.py is we can let it generate and install an executable for us instead of hard coding one at dwh-migration-tools/bin.dwh-migration-client. The setup.py generated executable will be put in the correct virtualenv or non-virtualenv location so that it's on the shell PATH and can be called without having to remember where the bin directory is. It will also respect pip's -e editable mode option. We tell setup.py to do this for us via console_scripts:
https://github.com/ajwelch4/dwh-migration-tools/blob/client-path-fix/client/setup.py#L36-L38
FInally, instead of telling users to edit the config files and input dir in the source tree directly, we provide them a simple init command that sets up the project scaffolding for them in a directory of their choosing:
dwh-migration-client init my_project

Our PWD now looks like this:
tree -L 1

.
├── dwh-migration-tools
├── my_project
└── venv

And if we cd into my_project we see:
tree

.
├── config.yaml
├── input
│   ├── 1.sql
│   ├── 2.sql
│   ├── google-examples.sql
│   └── subdir
│       └── 3.sql
├── macros.yaml
└── object_mapping.json

Executing the tanslation job is done via:
dwh-migration-client translate

or
dwh-migration-client translate --config config.yaml --input input --output output -m macros.yaml -o object_mapping.json

or some variant thereof.

Here are my proposed changes to this branch:
xiashang0624/dwh-migration-tools@client-path-fix...ajwelch4:client-path-fix
First, we should instruct the user to use a virtualenv to prevent dependency conflicts between our tool and the tools installed at the system-level by the system package manager. This goes without saying to Python folks as it's just good hygiene but not all of our customers will know this. It's particularly important in our case because we are asking our users to install gcloud and this is usually installed at the system-level. gcloud is written in Python and likely uses many of the same Google Cloud Python libs that we are and we can't count on users having the same version of gcloud and its transitive dependencies installed as we do. Most folks use Pipenv or Poetry nowadays to manage virtualenvs and store them in ~/.local/share/virtualenvs but the simplest way to convey the recommendation is to show the old fashioned way:
python3 -m venv venv
source venv/bin/activate

Pip, Pipenv and Poetry would also take care of downloading our tool if we were publishing it to PyPI but we are not so git clone it is:
git clone git@github.com:google/dwh-migration-tools.git

At this point, our PWD looks like this:
tree -L 1

.
├── dwh-migration-tools
└── venv

If the user doesn't plan on editing the tool and just wants to install it and get going:
pip3 install ./dwh-migration-tools/client

The above will invoke pips dependency resolver to install a set of concrete dependencies from the set of abstract dependencies listed in setup.py:
https://github.com/ajwelch4/dwh-migration-tools/blob/client-path-fix/client/setup.py#L23-L33
Using abstract dependencies in setup.py will allow us to support library use cases and PyPI distribution someday if we so choose. I would recommend tests before choosing so ;-). PSO is already investigating how to string together this tool, DTS, DVT, etc. so the library use case may be useful.
If the user wants to ensure they get the same exact concrete dependencies as us, they can run:
pip3 install -r ./dwh-migration-tools/client/requirements.txt
pip3 install ./dwh-migration-tools/client

Re: populating requirements.txt. The Python world has largely moved to lock files for concrete dependency management and Pipenv and Poetry would generate a lock file for us from the abstract dependencies. Instead, we will generate requirements.text from the abstract dependencies in setup.py by running:
make requirements

If the user wants to edit the tool, they can install it in editable mode using pip's -e option:
pip3 install -e ./dwh-migration-tools/client

Another benefit of using setup.py is we can let it generate and install an executable for us instead of hard coding one at dwh-migration-tools/bin/dwh-migration-client. The setup.py generated executable will be put in the correct virtualenv or non-virtualenv location so that it's on the shell PATH and can be called without having to remember where the bin directory is. It will also respect pip's -e editable mode option. We tell setup.py to do this for us via console_scripts:
https://github.com/ajwelch4/dwh-migration-tools/blob/client-path-fix/client/setup.py#L36-L38
setup.py lives in the client root directory. All other Python files have been moved into the dwh_migration_client package. It's pretty standard for the root directory of a Python project to only contain setup.py, markdown files, config files for formatters, linters, type checkers etc.
Finally, instead of telling users to edit the config files and input dir in the source tree directly, we provide them a simple init command that sets up the project scaffolding for them in a directory of their choosing:
dwh-migration-client init my_project

Our PWD now looks like this:
tree -L 1

.
├── dwh-migration-tools
├── my_project
└── venv

And if we cd into my_project we see:
tree

.
├── config.yaml
├── input
│   ├── 1.sql
│   ├── 2.sql
│   ├── google-examples.sql
│   └── subdir
│       └── 3.sql
├── macros.yaml
└── object_mapping.json

Executing the translation job is done via:
dwh-migration-client translate

or
dwh-migration-client translate --config config.yaml --input input --output output -m macros.yaml -o object_mapping.json

or some variant thereof.

Thanks for the review and I learnt a lot!
These are great suggestions and personally I REALLY like all of them from the perspective of engineering excellency.
First, I don't think this PR is conflicting with your suggestions so I would recommend to handle them in a following PR, unless you disagree, of course.
Here are my additional thoughts:
If the target audiences of this tool are experienced Python engineers, we should just do everything you suggested.
But I believe we are trying to make this tool so simple that people with almost no Python or coding experience can pick it up and run a translation job within just a few minutes. For this reason, I think we should mention virtualenv at the bottom of README in a "Common Questions" or "Recommendations" section instead of in the beginning.
The current basic user journey has 4 simple steps: people download this package from GH -> go to the root (dwh-migration-client) -> pip3 install requirements -> run the tool.
After a successful run, user should be happy and they can keep reading the rest of the doc about other features we provide.
I am not in favor of adding additional steps in the basic user journey. But I think we probably could replace "pip3 install client/requirements.txt" with "pip3 install dwh-migration-tools/client" after adding a setup.py file (in a follow-up PR).
And about "after user installs ./dwh-migration-tools/client, they can just call dwh-migration-tools from any directory without memorizing the path to the bin".  I don't think this will help with the issue this PR is trying to solve about the default path to the config files.  For example, if the user runs dwh-migration-tools from Users/xxx/Desktop, they still need to override the path to "config.yaml", "input", etc, right?  So I think this PR is still needed.  What do you think?
And about the logic of macros pre/post processing, in the new structure, do you think if we could also move macro_processor.py to the config folder with the other config files?  This is because we also expect users to change that python file, e.g., maybe they want to use a different postprocessing logic.  And this is for advanced users. They probably can figure out what file to change so it's not a high-priority.

@xiashang0624 Thanks for the detailed response! I agree with most of it. Much of this can be moved to follow-on PRs as suggested, so I've broken out my response below accordingly.
This PR
Checking PWD is repo root if CLI arg defaults are used

And about "after user installs ./dwh-migration-tools/client, they can just call dwh-migration-tools from any directory without memorizing the path to the bin".  I don't think this will help with the issue this PR is trying to solve about the default path to the config files.  For example, if the user runs dwh-migration-tools from Users/xxx/Desktop, they still need to override the path to "config.yaml", "input", etc, right?  So I think this PR is still needed.  What do you think?

I don't think the tool should check what directory it is in. I think it should validate the existence of the paths passed as CLI args (either by default or explicitly).
I don't know of any other tool that requires the user to be in a specific directory because of default settings. Every other tool I know just assumes the user is aware of the defaults and validates that they exist if they are file paths.
Also, the check as currently implemented isn't foolproof:
    if not os.getcwd().endswith(REPO_ROOT_DIR):
        if not args.config:
            print("Warning: you are not running the binary from the repo's root dir \"%s\" and you did not "
                  "pass a config file path through \"--config\". Trying to use the default config path \"%s\"."
                  % (REPO_ROOT_DIR, DEFAULT_CONFIG_PATH))
        if not args.input:
            print("Warning: you are not running the binary from the repo's root dir \"%s\" and you did not "
                  "pass an input path through \"--input\". Trying to use the default input path \"%s\"." %
                  (REPO_ROOT_DIR, DEFAULT_INPUT_PATH))
        if not args.output:
            print("Warning: you are not running the binary from the repo's root dir \"%s\" and you did not "
                  "pass an output path through \"--output\". Trying to use the default output path \"%s\"." %
                  (REPO_ROOT_DIR, DEFAULT_OUTPUT_PATH))

If someone does git clone git@github.com:google/dwh-migration-tools.git dwh-migration-tools-2, the warning is wrong. Also, although less likely, the lack of warning is wrong if someone runs the tool from a directory named dwh-migration-tools that is not the repo.
Here's a proposed implementation of simply validating the CLI args that builds on this branch: xiashang0624/dwh-migration-tools@client-path-fix...ajwelch4:aj-validate-paths.
Follow-on PRs
User journey: getting started

But I believe we are trying to make this tool so simple that people with almost no Python or coding experience can pick it up and run a translation job within just a few minutes. For this reason, I think we should mention virtualenv at the bottom of README in a "Common Questions" or "Recommendations" section instead of in the beginning.
The current basic user journey has 4 simple steps: people download this package from GH -> go to the root (dwh-migration-client) -> pip3 install requirements -> run the tool.
After a successful run, user should be happy and they can keep reading the rest of the doc about other features we provide.
I am not in favor of adding additional steps in the basic user journey. But I think we probably could replace "pip3 install client/requirements.txt" with "pip3 install dwh-migration-tools/client" after adding a setup.py file (in a follow-up PR).

I am basically in agreement with everything above. In particular, I agree mentioning virtualenv is overkill. Just thought I should point out not doing this will likely cause issues some day. I've removed mention of it from the docs completely.
Where I disagree slightly is I think the user journey should be the following:
These 2 commands only need to be run once (and they would be 1 command if we distributed to PyPI):
git clone git@github.com:google/dwh-migration-tools.git
pip3 install dwh-migration-tools/client

These 3 commands are the main user journey for a translation project:
dwh-migration-client init my_project
cd my_project
dwh-migration-client translate

This is a workflow that is very common. For example DBT, which is arguably one of the most popular CLI tools used by non-programmers who only know SQL, uses this workflow.
The tradeoff as I see it is the user has to copy and paste one additional command (dwh-migration-client init my_project) but what they get is a clean, unintimidating project scaffolding. And they can create additional isolated translation projects without having to git clone and pip install each time.
I have a proposed implementation I will submit as a separate PR. See the sequential list of follow-on PR diffs below.
User journey: macro_processor.py

And about the logic of macros pre/post processing, in the new structure, do you think if we could also move macro_processor.py to the config folder with the other config files?  This is because we also expect users to change that python file, e.g., maybe they want to use a different postprocessing logic.  And this is for advanced users. They probably can figure out what file to change so it's not a high-priority.

This is a good point. We could just tell the user to fork the repo but that is more work for the user. macro_processor.py is basically a plugin. The user may want to plug in other types of processors to process esoteric BI/ETL tool exports for example. I've already started working on processor functionality similar to the ksh ingester and inplace writer from the CW binary. I think we should create a Processor abstract base class (or Protocol but I don't think they are supported on all versions of Python 3.0+) that the user can inject implementations of into the client.
Having all the file handling logic in MacroProcessor makes this difficult. I think MacroProcessor should be limited to the functionality of MapBasedExpander more or less. The file handling logic should be abstracted away into a ProcessorPipeline class. ProcessorPipeline would take a list of Processor implementations.
I have a proposed implementation. It uses a processors.yaml file to configure ProcessorPipeline. However, given each processor will likely require vastly different config, a Python file may be better than a YAML file. I will submit the proposed implementation as a separate PR. See the sequential list of follow-on PR diffs below.
Docs
We may have to update the cloud.google.com docs if we take this approach? I'm happy to do that.
Sequential list of follow-on PR diffs

Formatter: ajwelch4/dwh-migration-tools@aj-validate-paths...ajwelch4:black-formatter
Linter: ajwelch4/dwh-migration-tools@black-formatter...ajwelch4:pylint
Type checker: ajwelch4/dwh-migration-tools@pylint...ajwelch4:mypy
User journey: getting started: ajwelch4/dwh-migration-tools@mypy...ajwelch4:user-journey-getting-started
User journey: plugins: ajwelch4/dwh-migration-tools@user-journey-getting-started...ajwelch4:user-journey-plugins

Thanks for the explanation and the proposed change. It looks good to me. I merged your commit with a few minor changes:

Refactored the args validation methods into a validation.py file.
Added a section in README with all options.
Deleted the Print usage message if no args are supplied step because bin/dwh-migration-tool is still the "quick-start" command in the current documentations.

For the follow-up PRs about user journey, the 'dwh-migration-client init my_project' step also makes sense to me now.  Thanks again for the explanation!  And thank you for drafting the PR!
About the abstract processors class, that was actually the initial plan ;) But we ended up with a simple macros_processor.py file because we (Shevek and I) expect people to read that file and we hope people (with minimum coding experience) is comfortable with modifying that python file directly.
But I really like the idea of providing other kinds of standard processing logics that users can just use to compose their own pre/post-processing pipeline.


remove process in favor of artisan, remove unused provider

Backpack\Base is no longer used.
Also I dunno why we use Process here, but Artisan can handle this call IMO.
I'v just tested and the file get's correctly published with Artisan. Any special reason for not using it here ?
Best,
Pedro

A new inspection was created.

The inspection completed: 1 new issues, 1 updated code elements

A new inspection was created.

The inspection completed: 1 updated code elements

Also I dunno why we use Process here, but Artisan can handle this call IMO.

Agreed. It's probably done with Process for consistency - since the other (non-artisan) commands use it.


Error: Cannot find module 'phantomjs-prebuilt'

Hello,
I'm trying to setup saleor-platform on my local dev machine. Sadly I'm getting error related to phantomjs-prebuilt in storefront_1 container.
I'm running Docker Desktop 3.3.2, Docker Engine 20.10.6 on M1 (Apple Silicon) Macbook Pro (but that shouldn't cause the issue...)
By following all of the steps from docs:
git clone https://github.com/mirumee/saleor-platform.git --recursive --jobs 3
cd saleor-platform
docker-compose build
docker-compose run --rm api python3 manage.py migrate
docker-compose run --rm api python3 manage.py collectstatic --noinput
docker-compose run --rm api python3 manage.py populatedb
docker-compose run --rm api python3 manage.py createsuperuser
docker-compose up

My error is:
storefront_1  | Using 1 worker with 2048MB memory limit
storefront_1  | ℹ ｢wds｣: Project is running at http://localhost:3000/
storefront_1  | ℹ ｢wds｣: webpack output is served from /
storefront_1  | ℹ ｢wds｣: Content not from webpack is served from /app
storefront_1  | ℹ ｢wds｣: 404s will fallback to /index.html
storefront_1  | ✖ ｢wdm｣: Error: Cannot find module 'phantomjs-prebuilt'
storefront_1  |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
storefront_1  |     at Function.Module._load (internal/modules/cjs/loader.js:562:25)
storefront_1  |     at Module.require (internal/modules/cjs/loader.js:692:17)
storefront_1  |     at require (internal/modules/cjs/helpers.js:25:18)
storefront_1  |     at Object.<anonymous> (/app/node_modules/svg2png/lib/svg2png.js:6:22)
storefront_1  |     at Module._compile (internal/modules/cjs/loader.js:778:30)
storefront_1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
storefront_1  |     at Module.load (internal/modules/cjs/loader.js:653:32)
storefront_1  |     at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
storefront_1  |     at Function.Module._load (internal/modules/cjs/loader.js:585:3)
storefront_1  | Browserslist: caniuse-lite is outdated. Please run next command `npm update`

I already tried several ways how to install phantomjs-prebuilt, but nothing worked. Adding RUN npm install<EMAIL_ADDRESS>--ignore-scripts into my saleor-storefront/Dockerfile.dev did not help, neither RUN npm install -g phantomjs-prebuilt --phantomjs_cdnurl=https://bitbucket.org/ariya/phantomjs/downloads --ignore-scripts or adding phantomjs-prebuilt into package.json.
Has anyone faced this issue? I'm pretty new to NodeJS as well as NPM errors.
Thanks.

Okay, I Think this got resolved by setting higher version of NODE package in Dockerfile.dev.