🧪 [linty] initial setup (preliminary)

This is a local recreation of #1027 by @racodond.

Looks good now! 🚀
Thanks for your help!
Now I will take a closer look at the "Quality Profiles".

Looks good now! 🚀 Thanks for your help!
Now I will take a closer look at the "Quality Profiles".

Hey @racodond.
Some questions...
I do not understand the configuration of the hierarchy. How to set a specific file as top module to?
I have problems understanding the "Only use allowed libraries: ieee.std_logic_1164, work" rule. There is just a not-allowed list but no allowed list. Does that mean that everything that is not on the list is allowed?
There are a lot of warnings regarding non-synthesizeable code elements. But this is just fine as they are used in the testbenches only. How do I exclude them from the synthesis-relevant checks?
Regarding the sonar-project.properties file: the web IDE seems to provide options for sonar.hdl.file.simulationPaths and sonar.hdl.topModule so those could be removed from that file, right?

Hey @stnolting,

I do not understand the configuration of the hierarchy. How to set a specific file as top module to?

You do not specify a file name but an entity name.
The top module should be set in two locations:

In hierarchy.ys file: Used by Tabby CAD to elaborate your design
At https://oss.linty-services.com/project/settings?id=neorv32&category=hdl, sonar.hdl.topModule property: Used by rules that apply differently on top module and sub modules.

I have problems understanding the "Only use allowed libraries: ieee.std_logic_1164, work" rule. There is just a not-allowed list but no allowed list. Does that mean that everything that is not on the list is allowed?

Yes, everything that is not listed is not allowed.
You can either work by blacklisting or whitelisting according to your needs. Either activate https://oss.linty-services.com/coding_rules?open=vhdl%3AVHDL153&rule_key=vhdl%3AVHDL153 or https://oss.linty-services.com/coding_rules?open=vhdl%3AVHDL312&rule_key=vhdl%3AVHDL312
Let me know if those two rules fulfill your needs or you'd like something else.

There are a lot of warnings regarding non-synthesizeable code elements. But this is just fine as they are used in the testbenches only. How do I exclude them from the synthesis-relevant checks?

Rules are tagged with 'simulation' or 'synthesis' or both depending on which context they apply.
https://oss.linty-services.com/coding_rules?languages=vhdl&q=synthe&open=vhdl%3AVHDL183 is tagged 'synthesis' only. So, if sonar.hdl.file.simulationPaths property at https://oss.linty-services.com/project/settings?id=neorv32&category=hdl is properly configured, you should not get issues on testbenches.
Note that properties overridden in sonar-project.properties take precedence. You do not have the same value in this file: https://github.com/stnolting/neorv32/pull/1029/files#diff-43ed9d31bea2a6d518d69836bcd1a8e6bd81bf4df96c4745792c220ca5aa549cR3
As it is a pattern, the value in the file is not correct. You should remove it and property defined through the web interface will apply.

Regarding the sonar-project.properties file: the web IDE seems to provide options for sonar.hdl.file.simulationPaths and sonar.hdl.topModule so those could be removed from that file, right?

Yes, properties should be defined through the web interface when possible.
Because when you use Linty in VS Code, properties are read from the server, not the sonar-project.properties file.
Hope it helps!

You do not specify a file name but an entity name. The top module should be set in two locations:

Right, my fault.
I would like to use the neorv32_top as top entity. This is set in the web interface (sonar.hdl.topModule) and also in linty_hierarchy.ys. So this should be fine, right? But what is the purpose of verific -L neorv32 -vhdl ... in linty_read.ys? Is there a command reference or verific somewhere?

Yes, everything that is not listed is not allowed. You can either work by blacklisting or whitelisting according to your needs.

Ah OK! I didn't there there are two rules - I only saw the not-allowed one.

Rules are tagged with 'simulation' or 'synthesis' or both depending on which context they apply.

:+1:

As it is a pattern, the value in the file is not correct. You should remove it and property defined through the web interface will apply.

Done.

Yes, properties should be defined through the web interface when possible.
Because when you use Linty in VS Code, properties are read from the server, not the sonar-project.properties file.

Sounds good!

Hope it helps!

Absolutely! Thank you very much.

I would like to use the neorv32_top as top entity. This is set in the web interface (sonar.hdl.topModule) and also in linty_hierarchy.ys. So this should be fine, right?

Right

But what is the purpose of verific -L neorv32 -vhdl ... in linty_read.ys?

Actually, -L neorv32 is not necessary and you can remove it.

Is there a command reference or verific somewhere?

Here's the doc: https://yosyshq.readthedocs.io/projects/yosys/en/latest/cmd/verific.html

Actually, -L neorv32 is not necessary and you can remove it.

:+1:

Here's the doc: https://yosyshq.readthedocs.io/projects/yosys/en/latest/cmd/verific.html

Thanks a lot!
There is still a problem with hierarchy elaboration:
[BugFinder] Hierarchy of your IP cannot be built.

sonar.hdl.topModule in the web interface is neorv32_top. linty_hierarchy.ys is hierarchy -top neorv32_top. What else is missing? 🤔

[BugFinder] Hierarchy of your IP cannot be built.

Logs should be saved in the action through: https://github.com/stnolting/neorv32/actions/runs/10971399621/workflow#L40
But it looks like they cannot be saved.
 So, could you please check if the .linty directory is created at the root of your directory and why it is not saved.
Once the workflow is fixed, the log of the elaboration of your design by Tabby CAD is available at .linty/bugfinder/logs/output.log.
Then, we'll know exactly why elaboration fails.

@stnolting : Did you manage to get the BugFinder output log?

Did you manage to get the BugFinder output log?

Unfortunately not yet. I have already checked the permissions, but everything seems to be correct. Maybe we should just add an ls -al to the job to see if the corresponding directory is created at all. 🤔

Unfortunately not yet. I have already checked the permissions, but everything seems to be correct. Maybe we should just add an ls -al to the job to see if the corresponding directory is created at all. 🤔

Yep, let's list files to check if we miss something obvious.

You need to add if: always() to the step to make it run even if the previous step failed:
- name: Test
  if: always()
  run: ls -al

Anyway. Linty fails since we have changed the top entity. In the previous (working) top entity all generics were defined/set, while in the current one (neorv32_top) some are undefined/unset. Maybe Linty/verific requires all generics to be defined for elaboration?

Indeed, for the elaboration with Verific, you have to provide values for generics with no default value.
You need to add some chparam properties in the hierarchy.ys file to set missing generic values. See https://doc.linty-services.com/scan.html#with-bugfinder-and-linter

run and uses cannot be used within the same job step.
I just tested to use a testbench as top entity - that also failed. Now I am using another wrapper (rtl/system_integration/neorv32_vivado_ip.vhd) as top entity and that seems to work...

run and uses cannot be used within the same job step.

Sure. You need a dedicated step for the ls.

I just tested to use a testbench as top entity - that also failed. Now I am using another wrapper (rtl/system_integration/neorv32_vivado_ip.vhd) as top entity and that seems to work...

Good!

@stnolting : I finally found out the issue about the .linty directory not being uploaded.
Here's the fix: https://github.com/Linty-Services/bugfinder-sample/commit/e283ca4aca80447f95f14fbeed4762c67950b7f7

Here's the fix:

Thank you very much!
I'm quite busy at the moment, but I'll come back to this ASAP (wekkend)! 🙈

Thank you very much!
I'm quite busy at the moment, but I'll come back to this ASAP (wekkend)! 🙈

Great!

Here's the fix:

Works like a charm! thank you very much!
I'm still trying to set the processor top as top for the elaboration. However, it seems like the tool cannot find that module:
ERROR: Module `neorv32_top' not found!

Do we need to specify the library somewhere in linty_hierarchy.ys?

Hmm, maybe this is just a code issue...
[00000.140434] 58. Executing hlp_synthesized_objects pass...
[[VHDL-1769;./../rtl/core/neorv32_top.vhd;25;5;25;20]] VERIFIC-ERROR [VHDL-1769] ./../rtl/core/neorv32_top.vhd:25: formal generic 'clock_frequency' has no actual or default value
[[VHDL-1067;./../rtl/core/neorv32_top.vhd;22;8;22;19]] VERIFIC-INFO [VHDL-1067] ./../rtl/core/neorv32_top.vhd:22: processing 'neorv32_top_default(neorv32_top_rtl)'
[[VHDL-1068;./../rtl/core/neorv32_top.vhd;22;8;22;19]] VERIFIC-INFO [VHDL-1068] ./../rtl/core/neorv32_top.vhd:22: processing of entity 'neorv32_top_default' failed
[00000.145408]
[00000.145410] 59. Executing command "echo synthesized_objects: passed >> /usr/src/.linty/bugfinder/logs/steps.log".
Build Hierarchy
[00000.145949]
[00000.145951] 60. Executing hlp_hierarchy pass...
[00000.145967]
[00000.145969] -- Executing script file `/usr/src/.github/linty_hierarchy.ys' --
[00000.145992]
[00000.145993] 61. Executing HIERARCHY pass (managing design hierarchy).
[00000.146172] ERROR: Verific static elaboration failed.

It makes sense that all generics of the top need to be defined 🤔

Hmm, maybe this is just a code issue...
[00000.140434] 58. Executing hlp_synthesized_objects pass...
[[VHDL-1769;./../rtl/core/neorv32_top.vhd;25;5;25;20]] VERIFIC-ERROR [VHDL-1769] ./../rtl/core/neorv32_top.vhd:25: formal generic 'clock_frequency' has no actual or default value
[[VHDL-1067;./../rtl/core/neorv32_top.vhd;22;8;22;19]] VERIFIC-INFO [VHDL-1067] ./../rtl/core/neorv32_top.vhd:22: processing 'neorv32_top_default(neorv32_top_rtl)'
[[VHDL-1068;./../rtl/core/neorv32_top.vhd;22;8;22;19]] VERIFIC-INFO [VHDL-1068] ./../rtl/core/neorv32_top.vhd:22: processing of entity 'neorv32_top_default' failed
[00000.145408]
[00000.145410] 59. Executing command "echo synthesized_objects: passed >> /usr/src/.linty/bugfinder/logs/steps.log".
Build Hierarchy
[00000.145949]
[00000.145951] 60. Executing hlp_hierarchy pass...
[00000.145967]
[00000.145969] -- Executing script file `/usr/src/.github/linty_hierarchy.ys' --
[00000.145992]
[00000.145993] 61. Executing HIERARCHY pass (managing design hierarchy).
[00000.146172] ERROR: Verific static elaboration failed.

It makes sense that all generics of the top need to be defined 🤔

You need to add some chparam properties in the hierarchy.ys file to set missing generic values. See https://doc.linty-services.com/scan.html#with-bugfinder-and-linter

I think this is quite ready. I'm very happy that all the modifications from this PR ended up in just a single file (the actual GitHub workflow file).
I'll try to adjust some of the rules before merging this (I do not want to start with a failed rating 😅). Here is the badge that we could add to the front page's status section:

While scrolling through the reports I found some strange reliability warning: "Remove forbidden operator "mod"."

This makes sense. mod is no good for synthesis. But in this case we are working with loop variable that are just constants at the end of the day (or at the end of synthesis). There is also a note regarding this:

Why is this a false positive? Did I miss something? Can't the checker identify this as a constant?

I think this is quite ready. I'm very happy that all the modifications from this PR ended up in just a single file (the actual GitHub workflow file). 🚀

Good!

I'll try to adjust some of the rules before merging this (I do not want to start with a failed rating 😅).

Sure

Here is the badge that we could add to the front page's status section:

OK

Why is this a false positive? Did I miss something? I mean the right operand is a constant, right? Can't the checker identify this as such?

mod operator was not in the list of allowed operators when right operand is a constant.
I made the description of the rule a bit more clearer. Let me know what you think and if it would suit your needs this way:

mod operator was not in the list of allowed operators when right operand is a constant.
I made the description of the rule a bit more clearer. Let me know what you think and if it would suit your needs this way:

I still don't get it 😅
In this case the right operand IS a constant and it IS a power of two as well. I cannot see any rule options to mark individual operators as valid when the previously mentioned conditions are fulfilled.

I still don't get it 😅

I fixed the rule to not raise an issue when mod has a right operand that is constant and power of 2. It'll be available in the next release. I also updated the rule description to make clear what the rule is checking.

Ahh ok :+1:
Sorry, then I just misunderstood you comment. 🙈

It'll be available in the next release.

Is there already a date for it?

It'll be available in the next release.

Is there already a date for it?

Our goal is for mid-November.

Hey @racodond!
How can I modify the quality gate requirements? For me, it seems they're fixed and cannot be altered. However, I would like to relax them so we do not have a "failed" status until the mod issue is fixed in the next release.

How can I modify the quality gate requirements? For me, it seems like they're fixed and cannot be altered at all. 🤔

I created a dedicated quality gate for your project: https://oss.linty-services.com/quality_gates/show/NEORV32
Feel free to update the requirements.

However, I would like to relax them so we do not have a "failed" status until the mod issue is fixed in the next release.

You should flag those issues as false positives.
See Bulk Change button.

I think we are ready to merge! 🚀
I will further investigate on the quality gates and design rules from the main branch - some failing rules are acceptable for now as this is still something like "work in progress" 😉

Oh wait, what happened here??

I think we are ready to merge! 🚀
I will further investigate on the quality gates and design rules from the main branch - some failing rules are acceptable for now as this is still something like "work in progress" 😉

Good news!

Oh wait, what happened here??
I did not change the token and it did not exceed the GitHub timeout 🤔

Did you generate a token with some expiration date?
Could you regenerate a token and try again?


Add Container PS

Add a little app 'Container PS' that list your Docker containers and allow you to manage them.

Oh, how you can see the image is corrupt?
But no problem, I'll make a new variant icon :)

Here we go :) It's updated


Migrate e2e test from control-plane

Description
Changes proposed in this pull request:

Migrate e2e test from control-plane
Add image builder workflow

Related issue(s)

/kind enchancement


Release plugin as a set of Unity Package Manager packages

Unity has a new package management system (UPM). It works pretty nice and has a decent number of advantages.

Library code is kept out of the Assets/ folder
If you open the Package Manager UI Unity will tell you when a package is out of date
Unity takes care of updating the package instead of needing t o do it (semi-)manually
There is no need to "Delete SteamVR folder and import" to update a package

I think it would be really nice to have UPM packages as an installation option for the SteamVR Unity Plugin.
Though the interaction system should probably be a separate package.

I've noticed a new branch UnityXRPluginis here for UPM, created by @zite. A great move! The version is not consistent with the main release yet (v2.6.x vs 1.0.0-preview.1), but maybe that is only for the beta stage.
I'm the creator of OpenUPM, a managed registry collects open-source UPM package with automatic build-pipelines. It monitors Git tags and builds new package releases. The platform is hosting more than 200 UPM packages at the time of writing. @ilterbilguven submits this package to our platform and brings me here.
However, all builds are failed because of lacking package.json in Git tags. To make it work, it requires creating a new Git tag based on the upm branch. e.g. whenever you create a GitHub release (from the master branch), you also create a Git tag named upm/x.y.z from the UnityXRPlugin branch. Then OpenUPM can take care of the rest to publish a new package release.
There're some articles discussed using CI to simplify the approach. But it can be a case-to-case thing to adopt to this repository.


Refactor props, using Object.assign() to override default props.

This makes it easier to express defaults in a single point.
What do you think?

👍
This looks good to me! Easier to read. Thanks for making a PR! I haven't checked directly, but the behavior is otherwise unchanged, right?

Thanks! Yes, the behavior is completely unchanged.


Need some help getting started

Hi
I tried and failed to branch / build / publish / install / test .. this crossplane provider ...
I have gotten the hang of the using part and installing the released provider.
However there seams to be a some part of of setup to run / build this local that I am missing
For some ctx, I am using M1 mac as daily driver. However i spun of a Ubuntu 22.04 as dev environment using multipass
I fork repo and and run 'make'
go build seams fine ...
make test, looks oki!
if I check docker image
$ docker images
REPOSITORY                                TAG               IMAGE ID       CREATED        SIZE
build-10484da4/provider-terraform-arm64   latest            1063824d22ee   5 hours ago    131MB
however now I would like to do a amd64 build and publish to a private repo so I can test deploy ?
Are there any instructions for this ?

Yes, so far so good!
But that only allows me to deploy what have already been released, and wont help me if I want to deploy and test my changes or some other changes (i.e. https://github.com/upbound/provider-terraform/pull/74) ?

make run(running binary locally) and make e2e(deploying Provider package to the local cluster) targets can help with local testing without dependency on local registries.

I have a local registry I just can't figure out how to make the 'build script' publish to my local registry instead ...

DOCKER_REGISTRY=xpkg.upbound.io/upbound/provider-terraform-staging make publish

something like that with your repo URL instead

Sry I must be dumb...
$ PLATFORM=linux_amd64 make
Works and I get an image for amd64, i think ...
REPOSITORY                                TAG               IMAGE ID       CREATED          SIZE
build-10484da4/provider-terraform-amd64   latest            3b377f9ac544   37 seconds ago   132MB
build-10484da4/provider-terraform-arm64   latest            64dfed450a07   4 minutes ago    131MB
however this still fails ?
$ PLATFORM=linux_amd64  DOCKER_REGISTRY=registry.company.dev/upbound/provider-terraform-staging make publish
19:08:45 [ .. ] Skipping image publish for xpkg.upbound.io/upbound/provider-terraform:v0.0.0-342.g9a22e93
Publish is deferred to xpkg machinery
19:08:45 [ OK ] Image publish skipped for xpkg.upbound.io/upbound/provider-terraform:v0.0.0-342.g9a22e93
19:08:45 [ .. ] Pushing package xpkg.upbound.io/upbound/provider-terraform:v0.0.0-342.g9a22e93
up: error: POST https://xpkg.upbound.io/v2/upbound/provider-terraform/blobs/uploads/: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:upbound/provider-terraform Type:repository] map[Action:push Class: Name:upbound/provider-terraform Type:repository]]
19:08:49 [FAIL]
make[1]: *** [build/makelib/xpkg.mk:105: xpkg.release.publish.xpkg.upbound.io/upbound.provider-terraform] Error 1
make: *** [build/makelib/common.mk:411: publish] Error 2


MAINT: Remove workaround

Let's see if we can get rid of this workaround yet. Probably not, but it's worth checking periodically.

Nope!


promptfoo view not able to connect to server via browser on 0.25.0

Hello,
I am currently using VSCODE dev container to install promptfoo. When I do a promptfoo view on 0.25.0, I run into the following on the browser:
Application error: a client-side exception has occurred (see the browser console for more information).
This is what I see on the browser console:
596-297f7ff4a0436e87.js:1 TypeError: Cannot read properties of null (reading 'toFixed')
at eo (891-0286b3010dc452ac.js:1:7166)
at ed (891-0286b3010dc452ac.js:1:8541)
at ri (fd9d1056-fba4b53a2f01213b.js:9:38931)
at im (fd9d1056-fba4b53a2f01213b.js:9:111754)
at oM (fd9d1056-fba4b53a2f01213b.js:9:90098)
at fd9d1056-fba4b53a2f01213b.js:9:89920
at oT (fd9d1056-fba4b53a2f01213b.js:9:89927)
at ov (fd9d1056-fba4b53a2f01213b.js:9:87565)
at oy (fd9d1056-fba4b53a2f01213b.js:9:86427)
at E (596-297f7ff4a0436e87.js:25:1389)

Have you run any evals / are you able to share the config of that eval and/or the contents of ~/.promptfoo/output/latest.json?

Release 0.25.1 contains a possible fix.


elasticache-controller - issue deleting CacheParameterGroup

Describe the bug
I was testing the elasticache-controller, and I have found that when I run kubectl delete on the following template (attached), all resources are deleted except for the CacheParameterGroup, whose delete fails because of the following condition:
Status:
  Ack Resource Metadata:
    Arn:               arn:aws:elasticache:us-west-1:*********:parametergroup:test
    Owner Account ID:  *********
    Region:            us-west-1
  Conditions:
    Last Transition Time:  2024-01-29T17:19:01Z
    Message:               Resource synced successfully
    Reason:
    Status:                True
    Type:                  ACK.ResourceSynced
    Message:               InvalidCacheParameterGroupState: One or more cache clusters are still members of this parameter group test, so the group cannot be deleted.
                           status code: 400, request id: b189ad54-88bc-4b4a-ae4d-1cb35c4f3e90
    Status:                True
    Type:                  ACK.Terminal

discussion on Slack: https://kubernetes.slack.com/archives/C0402D8JJS1/p1706552666549969
Steps to reproduce
Apply the attached template, then delete
Expected outcome
Expected the CacheParameterGroup CRD to be deleted
Environment

Kubernetes version
Using EKS (yes/no), if so version?
yes, v1.27.0
AWS service targeted (S3, RDS, etc.)
ElastiCache

template:
delete.txt

I just had the same issue. I'm using it inside a helm chart, so when I try to delete the chart the ParameterGroup gets stuck, I think is because by the moment it tries to delete it, the ReplicationGroup is still active. After the ReplicationGroup is gone I tried to delete it manually with no success.

@davejfranco We're working on fixing some flaky tests/security patches first. We'll dive into this one ASAP.


Copy over original input files

Sometimes things go wrong, so it would be useful to copy over the fasta file in the in folder into the out folder and append an .original
For example in/test.fasta --> out/test/test/test.fasta.original
Or something similar.

Closed in https://github.com/ajasja/af2slurm/pull/9


Time Varying Media Contribution Notebook

Description

Related Issue

[ ] Closes #
[ ] Related to #

Checklist

[x] Checked that the pre-commit linting/style checks pass
[ ] Included tests that prove the fix is effective or that the new feature works
[ ] Added necessary documentation (docstrings and/or example notebooks)
[ ] If you are a pro: each commit corresponds to a relevant logical change

Modules affected

[x] MMM
[ ] CLV

Type of change

[ ] New feature / enhancement
[ ] Bug fix
[x] Documentation
[ ] Maintenance
[ ] Other (please specify):

📚 Documentation preview 📚: https://pymc-marketing--778.org.readthedocs.build/en/778/

pre-commit.ci autofix

Indeed but a little, real intercept is 6. The recover 5.8

View entire conversation on ReviewNB

Good idea!

View entire conversation on ReviewNB

Can you experiment with the drop_first argument in the HSGP class? We might have redundant intercepts.

View entire conversation on ReviewNB

@cetagostini, very cool notebook! I left some comments :)

+1

View entire conversation on ReviewNB

Let's add a new line (or combine lines) in the README to talk about this feature

pre-commit.ci autofix

Done!

View entire conversation on ReviewNB

Done!

View entire conversation on ReviewNB

Done!

View entire conversation on ReviewNB

done!

View entire conversation on ReviewNB

pre-commit.ci autofix

pre-commit.ci autofix

pre-commit.ci autofix

Let's add a new line (or combine lines) in the README to talk about this feature

What do you think @cetagostini

pre-commit.ci autofix


[identity] Enable browser customization for web view

Packages impacted by this PR
@azure/identity
Issues associated with this PR
Fixes https://github.com/Azure/azure-sdk-for-js/issues/25277
Checklists

[x] Added impacted package name to the issue description
[ ] Added a changelog (if necessary)

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-identity


Auth and debug fail

I enabled debug to see why I wasn't receiving notifications and the following is the result when I create a dummy account to fire off a notification:
[2017-07-14 16:01:16 +0000] info [whostmgr5] wwwacct::Notify Notification =>
***Redacted***, ***Redacted*** via XMPP [eventimportance => Low (3)]
ERROR: Debug filehandle could not be opened.
        Debugging disabled.
       (Operation not permitted)
ERROR: Debug filehandle could not be opened.
        Debugging disabled.
       (Operation not permitted)
[2017-07-14 16:01:17 +0000] warn [whostmgr5] Failed to send notification of type “XMPP”: One or more notification attempts failed. Details below:
Error: XMPP authentication failed: $VAR1 = 'No such file or directory';

Error: XMPP authentication failed: $VAR1 = '';
 at /usr/local/cpanel/Cpanel/iContact.pm line 547.
	Cpanel::iContact::__ANON__(__CPANEL_HIDDEN__...) called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 122
	Try::Tiny::try(CODE(0x7be6208), Try::Tiny::Catch=REF(0x7be6c28)) called at /usr/local/cpanel/Cpanel/iContact.pm line 548
	Cpanel::iContact::_send_notifications(HASH(0x7072298), HASH(0x70c5928), ARRAY(0x7bde980)) called at /usr/local/cpanel/Cpanel/iContact.pm line 502
	Cpanel::iContact::icontact("subject", "New account: omgtest (omgtest.tld)", "html_related", ARRAY(0x78660e0), "x_headers", HASH(0x7c2e500), "event_name", "Notify", ...) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 504
	Cpanel::iContact::Class::_todo_inside_daemon(Cpanel::iContact::Class::wwwacct::Notify=HASH(0x71e9a70)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 332
	Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 80
	Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 101
	eval {...} called at /usr/local/cpanel/3rdparty/perl/524/lib64/perl5/cpanel_lib/Try/Tiny.pm line 92
	Try::Tiny::try(CODE(0x71e9c08), Try::Tiny::Catch=REF(0x6b6bf18)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 85
	Cpanel::iContact::Class::__ANON__() called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 64
	eval {...} called at /usr/local/cpanel/Cpanel/ForkAsync.pm line 64
	Cpanel::ForkAsync::do_in_child(CODE(0x71e9b60)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 87
	Cpanel::iContact::Class::_do_in_daemon(CODE(0x71e9ae8)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 334
	Cpanel::iContact::Class::send(Cpanel::iContact::Class::wwwacct::Notify=HASH(0x71e9a70)) called at /usr/local/cpanel/Cpanel/iContact/Class.pm line 323
	Cpanel::iContact::Class::new(__CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, __CPANEL_HIDDEN__, ...) called at /usr/local/cpanel/Cpanel/Notify.pm line 79
	Cpanel::Notify::__ANON__() called at /usr/local/cpanel/Cpanel/Notify.pm line 152
	Cpanel::Notify::_notification_backend("wwwacct::Notify", "No status set", 1, CODE(0x700a508)) called at /usr/local/cpanel/Cpanel/Notify.pm line 81
	Cpanel::Notify::notification_class("class", "wwwacct::Notify", "application", "wwwacct::Notify", "constructor_args", ARRAY(0x700a0b8)) called at /usr/local/cpanel/Whostmgr/Accounts/Create.pm line 1437
	Whostmgr::Accounts::Create::_wwwacct("no_cache_update", 0, "featurelist", "default", "hascgi", "n", "max_defer_fail_percentage", "unlimited", ...) called at /usr/local/cpanel/Whostmgr/Accounts/Create.pm line 1951
	Whostmgr::Accounts::Create::__createaccount("dkim", 1, "quota", "unlimited", "contactemail", "", "pkgname", "", ...) called at /usr/local/cpanel/Whostmgr/Accounts/Create.pm line 2001
	Whostmgr::Accounts::Create::_createaccount("dkim", 1, "quota", "unlimited", "contactemail", "", "pkgname", "", ...) called at whostmgr/bin/whostmgr5.pl line 753
	main::wwwacct() called at whostmgr/bin/whostmgr5.pl line 228

Edit: make test and make install worked without incident when installing

So I changed up the logging path to /tmp/xmpp.log and I'm getting:
XML::Stream: Read: buff(<failure xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><not-authorized/></failure>)
XMPP::Conn: AuthSASL: Authentication failed.

When I use the same username and password (as configured in cPanel) in Adium, I can connect no issue.

So I was able to get around the auth issues by changing:
my @result = $xmpp_conn->AuthSend(
To:
my @result = $xmpp_conn->AuthIQAuth(
....to avoid SASL issues.
Still getting:
Error: XMPP authentication failed: $VAR1 = 'No such file or directory';

Error: XMPP authentication failed: $VAR1 = '';
 at /usr/local/cpanel/Cpanel/iContact.pm line 547.

...the number of which those Error: lines occur is directly related to the number of recipients

Sorry for not seeing this one a when it was initially filed. Maybe my email filtering is too aggressive.
Will try to investigate this soon.

No worries - if it helps we are trying to auth through to our ejabberd servers...

My server is also running ejabberd. I saw similar issues while building this related to not initially passing in the 'component name', which is why I added that as an option. You'll get mysterious auth failures in Net::XMPP when not passing in a component name that matches what ejabberd thinks the node's hostname is. The only way I got reliable debugging info to realize this was by upping the loglevel in ejabberd then tailing the log on the server (Net::XMPP's debugging is next to useless). Oddly, for me there wasn't even any misconfiguration on the server side -- Net::XMPP just wasn't sending in the correct hostname without also sending in the component name. Supposedly this doesn't happen with other XMPP servers, but who knows?
Anyways, let me know if that helps. Otherwise, is this a public XMPP server (can I register an account there so that I can send messages to myself and further debug the issue)?

It is a private server but its federated.
Updated my primary XMPP.pm to the latest and enabled debugging which now does give some further details:
[2017-09-08 14:56:20 +0000] warn [suspendacct] Failed to send notification of type “XMPP”: One or more notification attempts failed. Details below:
Error: XMPP connection failed: $VAR1 = {
          'node' => bless( {
                             'TAG' => 'stream:error',
                             'CHILDREN' => [
                                             bless( {
                                                      'RAWXML' => [],
                                                      'TAG' => 'host-unknown',
                                                      'ATTRIBS' => {
                                                                     'xmlns' => 'urn:ietf:params:xml:ns:xmpp-streams'
                                                                   }
                                                    }, 'XML::Stream::Node' )
                                           ],
                             'RAWXML' => []
                           }, 'XML::Stream::Node' ),
          'type' => 'host-unknown'
        };
 at /usr/local/cpanel/Cpanel/iContact.pm line 548.

I've set component name to the FQDN of the host... SRV are good we are fully federated

ejabberd side:
2017-09-08 15:06:44.107 [debug] <0.16498.12>@ejabberd_receiver:process_data:284 Received XML on stream = <<"<?xml version='1.0'?><stream:stream version='1.0' xmlns:stream='http://etherx.jabber.org/streams' xmlns='jabber:client' to='mycpanelhostname.tld' from=mycpanelhostname.tld' xml:lang='en' >">>
2017-09-08 15:06:44.107 [debug] <0.16499.12>@ejabberd_c2s:send_text:1924 Send XML on stream = <<"<?xml version='1.0'?><stream:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='13464868883747332834' from='rawdomain.tld' version='1.0' xml:lang='en'>">>
2017-09-08 15:06:44.107 [debug] <0.16499.12>@ejabberd_c2s:send_text:1924 Send XML on stream = <<"<stream:error><host-unknown xmlns='urn:ietf:params:xml:ns:xmpp-streams'></host-unknown></stream:error>">>
2017-09-08 15:06:44.107 [debug] <0.16499.12>@ejabberd_c2s:send_text:1924 Send XML on stream = <<"</stream:stream>">>

and cpanel side debug level 2:
XML::Stream: Send: (<response xmlns='urn:ietf:params:xml:ns:xmpp-sasl'></response>)
XML::Stream: Process: block(0)
XMPP::Conn: AuthSASL: haven't authed yet... let's wait.
XMPP::Conn: Process: timeout(1)
XML::Stream: Read: buff(<failure xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><not-authorized/></failure>)
XML::Stream: Node: _handle_element: sid(2118189179703330220) sax(XML::Stream::Parser=HASH(0x535ba98)) tag(failure) att( xmlns urn:ietf:params:xml:ns:xmpp-sasl )
XML::Stream: Node: _handle_element: check( -1 )
XML::Stream: Node: _handle_element: sid(2118189179703330220) sax(XML::Stream::Parser=HASH(0x535ba98)) tag(not-authorized) att( )
XML::Stream: Node: _handle_element: check( 0 )
XML::Stream: Node: _handle_close: sid(2118189179703330220) sax(XML::Stream::Parser=HASH(0x535ba98)) tag(not-authorized)
XML::Stream: Node: _handle_close: check( 1 )
XML::Stream: Node: _handle_close: check2( 0 )
XML::Stream: Node: _handle_close: sid(2118189179703330220) sax(XML::Stream::Parser=HASH(0x535ba98)) tag(failure)
XML::Stream: Node: _handle_close: check( 0 )
XML::Stream: Node: _handle_close: check2( -1 )
XML::Stream: Process: block(0)
XMPP::Conn: AuthSASL: Authentication failed.

I know for a fact my auth is correct, ive used the credentials to login via adium

Definitely seems like I'm on the right track RE the component name issue I referenced since the server side seemed to indicate 'host-unknown'. There just has to be some other piece of the puzzle that causes these failures that I'm missing. Hopefully the additional details here should allow me to tweak my testing ejabberd server to simulate the failures and work on a fix, probably this weekend.

Just as a note, if I leave component blank, I no longer get host unknowns, I just get auth failures.
My last reply with level 2 debugging is with the plugin using SRV records (no hardcoded component)
What version of ejabberd are you running (we're on ejabberd-16.09) and perhaps what necessary config do you have?

16.09-4 is my ejabberd version. As to the "necessary" configuration, I wouldn't know. I use c2s with starttls on port 5222, but I'd figure that's pretty standard. My auth method is set to 'internal'. Nothing I'd figure that is out of the ordinary.

Thats all the same here as well... I'll keep an eye on the project, maybe something silly...
Thanks for the feedback, if you're on IRC (I'm on freenode) we could collab a bit

I can successfully reproduce the issue now after fiddling with things a bit. Not quite sure what I did yet, but I should be able to get to the bottom of it and push a fix up soon :D

Whatever this is, it appears to end with a socket error after which my Jabber server restarts (had to set loglevel to '5' to get this):
2017-09-08 18:05:58.311 [debug] <0.513.0>@ejabberd_socket:send:184 Error in fast_tls:send: {error,einval}
Otherwise, I've corrected the error handling -- it at least properly gives you the less nonsensical error message of 'not-authorized' now. From what I can tell, it is getting a request for auth , which the server responds with  and we send , which seems right, but then it repeats a challenge a second time, which we issue a blank  to. At that point the server returns a failure notice. Ugh. Will continue investigation this weekend.

Apparently the systemd service definition I had for ejabberd was acting up and sending SIGTERM to it over and over again. Was a red herring.
Real issue seems to be related to it preferring to use DIGEST-MD5 even though it's obsolete. Since I'm preferring STARTTLS anyways, using PLAIN or SCRAM would be fine, I think... though Net::XMPP doesn't appear to have any accessors for indicating your preference there. Gonna have to dig through Authen::SASL a bit I think.

Confirmed that this is a bug, though I can't be sure whether it is with Net::XMPP or ejabberd. In any case, when Net::XMPP encounters failures from an auth method, it does not attempt to use any of the other advertised login methods (like PLAIN), as setting 'disable_sasl_mechanisms: "DIGEST-MD5"' in my ejabberd.yml configuration file fixes the issue on my server. The way I intend to handle this is to rewrite the symbols in Net::XMPP local to the _send function in order to have it fall back to using PLAIN when it fails.
From what I can tell, libpurple issues a self-terminating XML tag as the second part of the challenge asked for here: https://wiki.xmpp.org/web/SASLandDIGEST-MD5
Whereas Net::XMPP issues a tag that is simply bereft of content on the interior. Either one of these should be OK to do, but it is the only visible difference in what gets sent to the server from libpurple versus Net::XMPP. This could be a parsing bug in ejabberd, not sure.


[Question] Create a CxxVector in Rust to return it to C++?

Hi,
thank you for this project, it is very very promising!
I'd like to return a CxxVector<u8> back to C++ from a Rust function, but can't see how to create a CxxVector in the first place.
#[cxx::bridge]
mod ffi {
    fn build() -> Result<CxxVector<u8>>;
}

What am I missing?

To do that, you should allocate some memory on the heap for the CxxVector content, then move it from the rust to the c++. Unfortunately, you can't do it straightforward, because CxxVector isn't trivially move constractible.
But you don't have to return CxxVector - just return rust::Vec, and then convert it to the CxxVector inside C++.
Simplifying my advice: return by value only language-specific types, and then convert it to the target language type inside the target language files.
For example:
 In Rust:
Create Vec<T>  =>  Pass it as Vec<T> to C++

In C++:
Get rust::Vec from Rust => Convert it by std::copy or std::transform to std::vector<T> => Use it

Hi @Geobert, Result<CxxVector<u8>> is not a type that you can possibly have in Rust, because std::vector has a move constructor and possibly internal pointers which are incompatible with how Rust does move semantics. You would need to use one of:

fn build(out: Pin<&mut CxxVector<u8>>) -> Result<()>; with a stack-allocated empty vector constructed by C++ for Rust to write the elements into, which is pinned and never moved by Rust

fn build() -> Result<UniquePtr<CxxVector<u8>>>; with a heap-allocated vector which you obtain in Rust via a call to an extern "C++" helper function that creates an empty UniquePtr<CxxVector<u8>> in C++ for Rust to fill and return back

fn build() -> Result<Vec<u8>>; using the Rust vector instead, which does not have internal pointers and so can be moved normally in Rust

I understand, thank you both!


Pre-release merge for v0.3.2

The primary purpose of this PR is to give the IT team a chance to get up to speed on how this service works.
The plain text (aka fulltext) extraction service was originally developed in early 2018 to support work on the overlap detection service, and in anticipation of work on the submission UI. In support of the submission UI v0.1-alpha, I circled back to bring the service up to speed with the latest requirements and deployment considerations.
The core functionality hasn't changed: pdf2text and pdftotext are used to grab plain text from the e-print PDF. This is stored in raw form as well as in a PSV format (required for some other backend services).
Edit: @mattbierbaum was responsible for the core functionality in the original implementation; my focus has been on the HTTP and async "glue".
If you've seen this code before, hopefully it will look a bit more thinned-down and easier to follow.
Things to look at:

Read the architectural overview for this service.
Take a look at these e2e tests to see how it is supposed to behave.
If you want to spin it up and try it out, read through the README on the develop branch.

Comments are welcome that suggest specific improvements, and that focus on:

Possible bugs in the implementation;
Areas where documentation or code are unclear.

Release notes - arXiv-NG - Version fulltext-0.3.2
        Task

[ARXIVNG-1620] -         Update deployment config for plain text service with shared volume

Release notes - arXiv-NG - Version fulltext-0.3.1
        Bug

[ARXIVNG-1069] -         Fulltext extraction worker does not discriminate between PDF and HTML (pending render) responses

        New Feature

[ARXIVNG-1602] -         Plain text extraction service should short-circuit if a task is already running/complete for a paper + extractor version

        Task

[ARXIVNG-1063] -         Upgrade fulltext extraction agent to use arxiv.base.agent

[ARXIVNG-1064] -         Remove modules/functionality from fulltext project that are redundant to arXiv base

[ARXIVNG-1065] -         Update fulltext service docs based on design discussions

[ARXIVNG-1066] -         Refactor fulltext store component to use s3

[ARXIVNG-1231] -         arxiv.auth integration for fulltext service

[ARXIVNG-1521] -         Refactor fulltext store component to use volume

        Story

[ARXIVNG-277] -         Fulltext service should generate and serve PSV text as a separate resource

I don't have any impression about how this interoperates with other services in arXiv Legacy or NG

For an overview, see https://arxiv.github.io/arxiv-submission-core/architecture.html#services-building-blocks

2. I continued to see MOUNTDIR-related exceptions in the JSON response until I chose some arXiv ID that was not 1802.00125 (any other valid ID works fine)

@mhl10 Any chance you have some log output for this?

after the app is started via docker-compose up, the very first POST request to the plaintext-api container always seems to fail with a ConnectionResetError.

@mhl10 I added some start-up checks (doing this for all services in the submission group) that should make it a bit more obvious when/whether the service is ready to handle requests. For both the API and the worker, you should see an INFO log message (level 20) "All upstream services are available; ready to start".
The big delay with this particular service is waiting for the DinD container to pull down the extractor image. I also tweaked the docker-compose and README a little to try to avoid having to pull the image every time you spin up the service group.
If the API is still waiting to start (because an upstream service is not ready), requests to the API should just hang. Here's an example of what start-up looks like (NOT to completion; the service is waiting for the worker to start, which depends on the extractor image being available):
plaintext-api           | [uWSGI] getting INI configuration from /opt/arxiv/uwsgi.ini
plaintext-api           | *** Starting uWSGI 2.0.18 (64bit) on [Wed May 15 13:33:00 2019] ***
plaintext-api           | compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-28) on 06 May 2019 17:26:23
plaintext-api           | os: Linux-4.9.125-linuxkit #1 SMP Fri Sep 7 08:20:28 UTC 2018
plaintext-api           | nodename: 4b47205309ca
plaintext-api           | machine: x86_64
plaintext-api           | clock source: unix
plaintext-api           | pcre jit disabled
plaintext-api           | detected number of CPU cores: 2
plaintext-api           | current working directory: /opt/arxiv
plaintext-api           | detected binary path: /root/.local/share/virtualenvs/arxiv-AI0WzCV4/bin/uwsgi
plaintext-api           | uWSGI running as root, you can use --uid/--gid/--chroot options
plaintext-api           | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
plaintext-api           | chdir() to /opt/arxiv/
plaintext-api           | your memory page size is 4096 bytes
plaintext-api           | detected max file descriptor number: 1048576
plaintext-api           | - async cores set to 100 - fd table size: 1048576
plaintext-api           | lock engine: pthread robust mutexes
plaintext-api           | thunder lock: disabled (you can enable it with --thunder-lock)
plaintext-api           | initializing 100 uGreen threads with stack size of 262144 (256 KB)
plaintext-api           | uwsgi socket 0 bound to TCP address :8000 fd 3
plaintext-api           | uWSGI running as root, you can use --uid/--gid/--chroot options
plaintext-api           | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
plaintext-api           | Python version: 3.6.5 (default, Apr 10 2018, 17:08:37)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
plaintext-api           | Python main interpreter initialized at 0x26e3fb0
plaintext-api           | uWSGI running as root, you can use --uid/--gid/--chroot options
plaintext-api           | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
plaintext-api           | python threads support enabled
plaintext-api           | your server socket listen backlog is limited to 100 connections
plaintext-api           | your mercy for graceful operations on workers is 60 seconds
plaintext-api           | mapped 2216128 bytes (2164 KB) for 100 cores
plaintext-api           | *** Operational MODE: async ***
plaintext-api           | application 15/May/2019:09:33:01 +0000 - arxiv.vault.manager - DEBUG: "New Vault connection at mock-vault://8200:http"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "await Storage"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "service Storage is available!"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "await CanonicalPDF"
mock-arxiv              | [pid: 14|app: 0|req: 1/1] <IP_ADDRESS> () {26 vars in 319 bytes} [Wed May 15 13:33:06 2019] HEAD / => generated 0 bytes in 6 msecs (HTTP/1.1 200) 2 headers in 78 bytes (0 switches on core 0)
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "service CanonicalPDF is available!"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "await Compiler"
mock-arxiv              | [pid: 13|app: 0|req: 1/2] <IP_ADDRESS> () {28 vars in 348 bytes} [Wed May 15 13:33:06 2019] GET /compiler/status => generated 0 bytes in 3 msecs (HTTP/1.1 200) 2 headers in 78 bytes (1 switches on core 0)
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "service Compiler is available!"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.factory - None - [arxiv:null] - INFO: "await fulltext.extract"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.extract - None - [arxiv:null] - DEBUG: "check connection to task queue"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.extract - None - [arxiv:null] - DEBUG: "connection to task queue ok"
plaintext-api           | application 15/May/2019:09:33:06 +0000 - fulltext.extract - None - [arxiv:null] - DEBUG: "waiting for task result"

One issue with the previous full text extraction was that some PDFs caused very long run times pdfminer (pdf2txt.py). I think the pdf for submission/1152695 was one of them.
https://arxiv-org.atlassian.net/browse/ARXIVDEV-2688
Consider finding some of these long running pdfs and including them in your tests. Matt might have a good lead on these.

One issue with the previous full text extraction was that some PDFs caused very long run times pdfminer (pdf2txt.py). I think the pdf for submission/1152695 was one of them.
https://arxiv-org.atlassian.net/browse/ARXIVDEV-2688
Consider finding some of these long running pdfs and including them in your tests. Matt might have a good lead on these.

Created ARXIVNG-2167 so that we don't lose track of this. Since we are doing the extraction asynchronously, it shouldn't hurt too bad if the extraction takes a little while.

So if we update pdftotext we do not necessarily need to regenerate fulltext that was generated by pdfminer (the preferred extraction application). On the other hand, if we update pdfminer, then we might want to reprocess everything (since we prefer the output from pdfminer).

That's an excellent point. We should be able to come up with a versioning strategy for the extractor as a whole that takes that into account.

The README.md has a TODO section at the top with one item being the refactoring of celery instantiation. Given all of the configuration work you've completed, I'm wondering if this work is complete, or if not whether these items are needed for alpha.
I seem to have successfully built the fulltext service with docker-compose, though with more red/errors than I was hoping to see in the build output.  I'll do more testing tomorrow.
Overall the README.md looks good.

The README.md has a TODO section at the top with one item being the refactoring of celery instantiation. Given all of the configuration work you've completed, I'm wondering if this work is complete, or if not whether these items are needed for alpha.

Put those in for v0.4. They aren't blocking for alpha, just want to signal a move toward more coherent configuration management for these API/worker combos.

This comment may be slightly outside the scope of this PR. I worked on the extraction code in the legacy system so I was taking a quick look at the core extractor/fulltext/fulltext.py/fulltext() extractor method. If I understand the new code correctly if pdfminer fails we then run pdftotext, and if this output looks ok we use this output text. If pdftotxt fails we try pdfminer with -A option. Is this accurate?
The pdfminer code is run inside a try. The pdftotext is not. Are there any failure cases for pdftotext?

After looking over the code I feel much more comfortable with the arxiv-fulltext service.
I did take some time to look back at the original fulltext processing in the legacy Perl-based arXiv system. The core algorithm appears to differ slightly from how the current production fulltext routine operates. This may be intentional. I'm wondering if the NG fulltext was tested over a large set of published papers and the fulltext output compared to the output produced by the legacy system. I also note the legacy routine is more verbose in logging which of the three extraction methods get applied to a particular paper. This may have been more useful when we were debugging several issues we were experiencing with this code but could be useful when it comes time to expand the number of extractions options. I sometimes wonder, for a fleeting moment, about whether the level of complexity is absolutely necessary as a single file handling fulltext extraction in the legacy system has been replaced by several technologies and more than one hundred and fifty files in the NG system.
Issues with my laptop environment dind/docker limited my ability to test the fulltext to the extent I had originally wanted.

I'm wondering if the NG fulltext was tested over a large set of published papers and the fulltext output compared to the output produced by the legacy system.

@mattbierbaum has been using this extraction routine on very large chunks of the corpus, and using the results to feed into work on the classifier, overlap detection, etc. The goal wasn't to precisely replicate the extraction of the classic system, but to produce extractions that were suitable for use in the classifier. So I'm pretty confident that we've at least done that.

I also note the legacy routine is more verbose in logging which of the three extraction methods get applied to a particular paper. This may have been more useful when we were debugging several issues we were experiencing with this code but could be useful when it comes time to expand the number of extractions options.

Created https://github.com/arXiv/arxiv-fulltext/issues/19 so we don't lose track of this -- good idea.

I sometimes wonder, for a fleeting moment, about whether the level of complexity is absolutely necessary as a single file handling fulltext extraction in the legacy system has been replaced by several technologies and more than one hundred and fifty files in the NG system.

That's a good concern to have. One thing that I would offer is that the intended scope of this service goes well beyond the legacy code that runs in the submission process right now. For example, this is intended to also support access to extractions of announced e-prints via the public API.

That's a good concern to have. One thing that I would offer is that the intended scope of this service goes well beyond the legacy code that runs in the submission process right now. For example, this is intended to also support access to extractions of announced e-prints via the public API.

I agree with you that the service is definitely supporting more functionality. I believe the issues I ran into while trying to test also prompted my concern about complexity.

I'll chime in here. What dataset were you running your tests on @DavidLFielding? I'm not entirely sure of the provenance of my fulltext - could have been from production, maybe a student had created it, it was on a separate server.

@mattbierbaum I was comparing NG code to ExtractText.pm code running on our production web nodes. The only difference I thought might matter, and I might not understand your code completely, is that the new code falls back to pdftotext when pdf2txt initially fails, whereas the old code tries pdf2txt -A and leaves pdftotext as the extraction option of last resort. It's possible this doesn't really matter but in the past we were alerted when we were generating lower quality fulltext using pdftotext when pdf2txt was able to produce higher quality text.
I had difficulties with my laptop setup so I could not complete the testing I wanted to. I was intending to test papers where the initial pdf2txt fails to determine which extractor app ultimately gets called to do the work.


SongInfo is not comparable

Try:
Stream<List<SongInfo>> _songStream =
    Stream.fromFuture(audioQuery.getSongs());
_songStream.listen((List<SongInfo> _songList) {
  _songList.sort();
});

Error thrown in my code:
I/flutter (23594): ══╡ EXCEPTION CAUGHT BY GESTURE ╞═══════════════════════════════════════════════════════════════════
I/flutter (23594): The following assertion was thrown while handling a gesture:
I/flutter (23594): type 'SongInfo' is not a subtype of type 'Comparable<dynamic>'
I/flutter (23594): Either the assertion indicates an error in the framework itself, or we should provide substantially
I/flutter (23594): more information in this error message to help you determine and fix the underlying cause.
I/flutter (23594): In either case, please report this assertion by filing a bug on GitHub:
I/flutter (23594):   https://github.com/flutter/flutter/issues/new?template=BUG.md
I/flutter (23594): When the exception was thrown, this was the stack:
I/flutter (23594): #0      ListMixin._compareAny (dart:collection/list.dart:341:31)
I/flutter (23594): #1      Sort._insertionSort (dart:_internal/sort.dart:69:36)
I/flutter (23594): #2      Sort._doSort (dart:_internal/sort.dart:58:7)
I/flutter (23594): #3      Sort.sort (dart:_internal/sort.dart:33:5)
I/flutter (23594): #4      ListMixin.sort (dart:collection/list.dart:335:10)
I/flutter (23594): #5      _PlayerState.onSet (package:stepslow/main.dart:379:15)
I/flutter (23594): #6      _PlayerState.build.<anonymous closure>.<anonymous closure> (package:stepslow/main.dart:1039:45)
I/flutter (23594): #7      _InkResponseState._handleTap (package:flutter/src/material/ink_well.dart:654:14)
I/flutter (23594): #8      _InkResponseState.build.<anonymous closure> (package:flutter/src/material/ink_well.dart:729:32)
I/flutter (23594): #9      GestureRecognizer.invokeCallback (package:flutter/src/gestures/recognizer.dart:182:24)
I/flutter (23594): #10     TapGestureRecognizer._checkUp (package:flutter/src/gestures/tap.dart:365:11)
I/flutter (23594): #11     TapGestureRecognizer.acceptGesture (package:flutter/src/gestures/tap.dart:312:7)
I/flutter (23594): #12     GestureArenaManager.sweep (package:flutter/src/gestures/arena.dart:156:27)
I/flutter (23594): #13     GestureBinding.handleEvent (package:flutter/src/gestures/binding.dart:222:20)
I/flutter (23594): #14     GestureBinding.dispatchEvent (package:flutter/src/gestures/binding.dart:198:22)
I/flutter (23594): #15     GestureBinding._handlePointerEvent (package:flutter/src/gestures/binding.dart:156:7)
I/flutter (23594): #16     GestureBinding._flushPointerEventQueue (package:flutter/src/gestures/binding.dart:102:7)
I/flutter (23594): #17     GestureBinding._handlePointerDataPacket (package:flutter/src/gestures/binding.dart:86:7)
I/flutter (23594): #21     _invoke1 (dart:ui/hooks.dart:263:10)
I/flutter (23594): #22     _dispatchPointerDataPacket (dart:ui/hooks.dart:172:5)
I/flutter (23594): (elided 3 frames from package dart:async)
I/flutter (23594): Handler: "onTap"
I/flutter (23594): Recognizer:
I/flutter (23594):   TapGestureRecognizer#e9d0d
I/flutter (23594): ════════════════════════════════════════════════════════════════════════════════════════════════════

Recommended patch:
class SongInfo implements Comparable<SongInfo> {

or maybe:
class DataModel implements Comparable<DataModel> {

instead of:
class SongInfo {

A possible workaround until a solution would be:
_songList.sort((a, b) => a.filePath.compareTo(b.filePath));

Closed as solved or closed as wontfix?


Consider providing an plugin abstraction for intents

The Android intent system is based on request codes, which means that distributing handling of responses among plugins that don't know about each other is problematic. https://github.com/flutter/flutter/issues/124309 is a recent example of this, and I've seen people ask before how they are intended to avoid it.
While there are reasonable mitigations (as noted in my comment), plugin authors have to know to do them, and all it takes is for two not to, and to instead assign their codes in sequence from 0 or 1 (or to both pick "clever" special values instead of random ones, and have the same clever idea), for those options to break when combined.
Since the underlying problem is that we don't know about the request so can't route the responses, we could provide a wrapper API for plugins to use for intents, which could handle it internally (e.g., by taking half the intent code size for a per-plugin, Flutter generated value, then letting each plugin use the other half-size for its codes in the wrapper API, then using the generated half to route responses).
A counterargument here would be that people would also have to know to use the wrapper instead of doing intents directly, but we could potentially deprecate the API to register the current listener.

FYI this is a good description of the problem: https://github.com/flutter/flutter/issues/124309#issuecomment-1498912693.

This would be similar in spirit to the ActivityResultRegistry in AndroidX, which supports its registerForActivityResult API.
A related problem that would be useful to solve, and that I think could be solved by the same new abstraction, is to interoperate with AndroidX's "activity result contracts". I'm thinking of https://github.com/flutter/plugins/pull/7043#issuecomment-1435136191 , where:

To use the Android photo picker, the Android docs recommend some pretty clean-looking code involving the PickVisualMedia activity result contract.

That contract encapsulates dealing with all the mess of falling back through four different intent types depending on what's available on the device.
It also provides type safety in interpreting the result: the application code passes a callback that takes a Uri? parameter.

But that sample code in the Android docs uses registerForActivityResult, and that's only available if your application's activity is an AppCompatActivity. Which isn't the case in Flutter.
So things were stuck, and it seemed like we wouldn't be able to take advantage of PickVisualMedia.

By digging into how this works, I found that it is possible to use an activity result contract like PickVisualMedia without registerForActivityResult and without an AppCompatActivity. The code even comes out pretty clean.
But it wasn't at all obvious how to do that, and I think most developers are not going to find it — I think it's not possible to learn from the docs, and you have to read the AOSP source.

It would be convenient if a new API we provided for this area offered a way to consume an ActivityResultContract, so that it became very straightforward to translate sample code from Android docs to use Flutter's APIs.
Here's Android's docs on how to start an activity for a result, all in terms of ActivityResultContract:
https://developer.android.com/training/basics/intents/result

handle it internally (e.g., by taking half the intent code size for a per-plugin, Flutter generated value, then letting each plugin use the other half-size for its codes in the wrapper API, then using the generated half to route responses).

Just as a point of comparison to some prior art, here's how AndroidX seems to handle it.
The class that powers the dispatch is ActivityResultRegistry:
https://android.googlesource.com/platform/frameworks/support/+/a67808214/activity/activity/src/main/java/androidx/activity/result/ActivityResultRegistry.java
and it just maintains a map from string keys to result codes and back. The result codes are generated randomly at register time, and it avoids values less than 2 ** 16.
Those string keys are generated in registerForActivityResult:
https://android.googlesource.com/platform/frameworks/support/+/a67808214/activity/activity/src/main/java/androidx/activity/ComponentActivity.java#854
with just a sequence number.
So in order to keep the mapping from result codes back to string keys, and from string keys back to callbacks, stable between launches of the app (which is important because if you started something memory-intensive like the camera, your app is probably getting killed while that runs and relaunched when it's done), the register calls have to happen in the same order.
To accomplish that, they rely on an admonition in the docs:

You must call registerForActivityResult() in the same order for each creation of your fragment or activity so that the inflight results are delivered to the correct callback.

plus I guess the fact that the most natural ways to write the code (at least in simple apps) will always do them in some fixed order anyway.

It would be convenient if a new API we provided for this area offered a way to consume an ActivityResultContract, so that it became very straightforward to translate sample code from Android docs to use Flutter's APIs.

I think this is key. Currently reading Android docs and then trying to convert what you've learned there to a Flutter plugin, is a bit of a headacre, due to how a flutter plugin is structured. Also want to point out that registerForActivityResult is part of the new Android Activity Result API (which is strongly recommended by Google over the old startActivityForResult approach).
I think it would be preferred to shy away from how FlutterEngineConnectionRegistry currently takes it upon itself to configure the plugin for every conceivable scenario (which breaks the Single Responsibility Principle in a big way).
I'd really prefer this responsibility be moved out of that class entirely, and do something similar to how Android handles it by basically allowing things to be registered/deregistered on the fly. ActivityResults, BroadcastReceivers etc.
(You can technically already do that with BroadcastListeners on the fly: ContextCompat.registerReceiver(context, MyBroadCastListener(), intentFilter, ContextCompat.RECEIVER_EXPORTED))
But I digress... In short, I'd love an easy way to use registerForActivityResult.
(I'm currently trying to wrap permissions results in a reusable way using Kotlin's coroutines, but it's a head scratcher... 😆)


Temporarily disable smart builds until bldr_channel issues are addressed

:nut_and_bolt: Description
By default, habitat looks in HAB_BLDR_CHANNEL and then stable when
attempting to find package dependencies. What we need to be able to do
is specify a list of channels (i.e. bldr-*,dev,stable).
Signed-off-by: Tom Duffield<EMAIL_ADDRESS>:+1: Definition of Done
:athletic_shoe: Demo Script / Repro Steps
:chains: Related Resources
:white_check_mark: Checklist

[ ] Necessary tests added/updated?
[ ] Necessary docs added/updated?
[ ] Code actually executed?
[ ] Vetting performed (unit tests, lint, etc.)?

I have discovered another workaround that does not require us to disable smart builds. Will get that deployed to Expeditor ASAP.


[DO NOT MERGE] Introduce Coroutine Context provider for ViewModel testability

DO NOT MERGE
In #655 we need to be able to control the execution of coroutines in a ViewModel in order to assert the expected "immediate" behaviour before the suspended operation would overwrite this value.
Since traditional test dispatchers provided defer to "immediate" execution using Dispatchers.Unconfined, this cannot be directly achieved.
This pull request, demonstrates how we can use a similar behaviour to provide a CoroutineContext instead of CoroutineDispatcher to test ViewModel's behaviour, the test has been written by @manuelvicnt in #655, I've only demonstrated here how to use TestCoroutineContext.
Adversely, I would actually recommend against this solution, since it introduces confusion as to whether one should use CoroutineContext or CoroutineDispatcher and does not clearly identify the nuances between them.
Furthermore, a pull request from @objcode proposes a better implementation of TestCoroutineContext, splitting it up into TestCoroutineDispatcher and accompanying TestCoroutineScope, properly respecting the hierarchical nature of coroutine execution.
It would probably be better to wait for this to become available in coroutine-test rather than going with this implementation, but I have no idea how long that might be.
Consideration
In our workflow we conside a pull request to be invalid if we don't provide a meme to accompany it, so here is mine 😄

Thanks for the contribution Ash. This is a good understanding of how the Coroutine Context works but unfortunately we're going to wait until there are some improvements in this space to get that test in place. The changes to make that happen introduce some inconsistencies that we prefer to avoid. Thanks for your time on this, it helped us understand what's needed to have that working. Closing the PR and hoping we can create one to solve the issue soon :)

I would absolutely agree that it's worth waiting for further improvements, especially with regard to structured concurrency, I'm glad this was helpful and I hopefully they won't be too long 😉


Fix outType on gitea dump

Force to output the dump file to use the given --type.
Resolves #17959

Beautifull, yet another CI error which I haven't seen before:
https://drone.gitea.io/go-gitea/gitea/48222/5/2
mpm ERR! code E503
npm ERR! 503 Service Unavailable - GET https://registry.npmjs.org/ansi-styles/-/ansi-styles-3.2.1.tgz

Should this be a backport?

yes please


The builtin service type symbol is incorrectly defined

Description:
$title
https://github.com/ballerina-platform/ballerina-lang/blob/068d219290b2661f03c325cbe29815860ebd8359/compiler/ballerina-lang/src/main/java/org/wso2/ballerinalang/compiler/semantics/model/SymbolTable.java#L202
Steps to reproduce:
Affected Versions:
OS, DB, other environment details and versions:
Related Issues (optional):

Suggested Labels (optional):

Suggested Assignees (optional):

This will be changed with the new service type proposal in SL. Hence this change is no longer needed.


Apply string intern for device String

The implementation of cacheString pool has the memory leak risk. We can use string intern instead of do it by ourself.

Coverage increased (+0.007%) to 67.74% when pulling ad923c12e1b21f0c46c91f5180a7090af25d4a74 on chengjianyun:jianyun/intern into 4e87d43d0bd18b30f96db9745bdef89c6ee1f4c1 on apache:master.

Hi, is there any detailed reason about the memory leak risk?

Hi, is there any detailed reason about the memory leak risk?

cachedDevicePool never remove its key which is not reasonable. For example, if we delete some storage groups or timeseries, some string would be never used but they are always kept in cachedDevicePool which cause memory leak.
For string intern, JVM takes care of remove the useless string objects.

For example, if we delete some storage groups or timeseries, some string would be never used but they are always kept in cachedDevicePool which cause memory leak.

Hi, it's true that we can't handle the useless string if we delete some devices. But the intern() method is not that fast[1][2] since the purpose of it is to save memory. By comparison, concurrentHashMap has a good performance.
I believe that the situation about deleting timeseries or storage group is relatively few. And we use deviceIndexMap when we initialize TsFileResource, when restarting or ingesting data, the performance seems more important. Maybe we need to make a balance here. Or we can delete useless string from cachedStringPool manually when time series is deleted.
What do you think?
[1] https://stackoverflow.com/questions/56203500/java-string-intern-use-hashtable-instead-of-concurrenthashmap
[2] https://www.fatalerrors.org/a/jvm-series-string.intern-performance-of.html

For example, if we delete some storage groups or timeseries, some string would be never used but they are always kept in cachedDevicePool which cause memory leak.

Hi, it's true that we can't handle the useless string now if we delete some devices. But the intern() method is not that fast[1][2] since the purpose of it is to save memory. By comparison, concurrentHashMap has a good performance. I believe that the situation about deleting timeseries or storage group is relatively few. And we use deviceIndexMap when we initialize TsFileResource, when restarting or ingesting data, the performance seems more important. Maybe we need to make a balance here. Or we can delete useless string from cachedStringPool manually when time series is deleted instead of just using intern(). What do you think?
[1] https://stackoverflow.com/questions/56203500/java-string-intern-use-hashtable-instead-of-concurrenthashmap [2] https://www.fatalerrors.org/a/jvm-series-string.intern-performance-of.html

@Alima777, about performance of  String.intern(), you can also refer  this blog. I have verified the conclusion and got similar result on JDK8. So, it's better to set the -XX:StringTableSize when go bad perfromance on intern() method. And the performance improved much since JDK11 event without  -XX:StringTableSize parameter.
BTW, memory leak is intolerable in any cases especially in a database server system.

BTW, memory leak is intolerable in any cases especially in a database server system in my opinion.

Hi, I understand your concerns. So what I said is we can delete useless string from cachedStringPool manually when time series is deleted to avoid memory leak. I didn't check the conclusion about memory footprint using WeakHashMap, maybe it stores more things than normal HashMap?

we can delete useless string from cachedStringPool manually when time series is deleted

As far as I know, we can do this by count references to each object like older verion JVM GC does. If we do like this, why don't let JVM help recycle the memory usage by use WeakHashMap? Or do you have other ideas?

WeakHashMap is 4~5 times thanintern(), is concurrentHashMap still the same?

The HashMap or ConcurrentHashMap implemented CachePool is wrong without explicit clear, so I didn't verify with them.

As far as I know, we can do this by count references to each object like older verion JVM GC does.

Hi, you should know we only use this cachedStringPool in limited situation. It's only used in TsFileResource. So count references maybe not needed, we just add it when initialize, and remove it when delete timeseries.

So count references maybe not needed, we just add it when initialize, and remove it when delete timeseries.

I don't think this way would work. As the strings from CacheStringPool are passed to different instances, you must know all of the references of a key are released then the CacheStringPool could remove the key. Otherwise, you would hold different instances for same string which will lose the meaning of CacheStringPool.
And for the class, it designs to provide ability like string intern, but has implementation defects. You can't restrict it only used to hold device or measurement path unless you name it something like DevicePathPool or MeasurementPathPool instead of CacheStringPool.
The two ways I can figure out are either use WeakHashMap<String, WeakRefernce<String>> instead of ConcurrentHashMap or  String.intern(). I prefer the later one in the PR.

memory leak is intolerable in any cases especially in a database server system in my opinion.

+1


SAMZA-1296 : Stand alone integration tests.

Changes
Brings up a test bed that contains embedded kafka broker and zookeeper to test the following scenarios.
A) Rolling upgrade of stream processors.
B) Reelection of leader upon failures.
C) Registering multiple processors with same processor id.

NOTE:
Some tests are commented out since zookeeper exceptions are swallowed in ZKJobCoordinator/ZKUtils.

@navina @sborya
Please take a look when you have a chance.
@navina Will create JIRA's for issues we discussed offline.

@navina
Thanks for your review.
A) I'm fixing the cause for commented out integration tests.
B) Let's add gradle build flag to disable the integration tests during each build and run only when it's turned on.
Let's land this patch after A, B goes in.


CAST function is not supported with bytea

CAST function is not supported with bytea
Environment:

eKuiper version : 1.9
Hardware configuration: 4 core/16 GB RAM
OS : Windows/Docker
Others:

What happened and what you expected to happen:
I am not able to create a rule with cast function as cast(col, "bytea") as per documentation https://ekuiper.org/docs/en/latest/sqls/functions/transform_functions.html#cast
Expected behavior: Should able to create a rule with cast(col, "bytea")
How to reproduce it (as minimally and precisely as possible):
When you try to create a rule using ekuiper manager (version: 1.9) or eKuiper API like sql statement as SELECT CAST(base64image,"bytea") FROM mqttstream
Anything else we need to know?:
eKuiper rule creation API gives 400 bad request with below error message:
invalid rule json: Parse SQL SELECT CAST(*,"bytea") FROM mqttstream error: Expect one of following value for the 2nd parameter: bigint, float, string, boolean, datetime..

Could you please provide me the ETA, if possible, when this will be available? or is there any short-term solution till this issue be fixed? I know that PR is ready already.

This will be included in 1.11 release planned in mid-August. We will also release a pre-release like 1.11.0-alpha.2 recently.


DatePicker not working properly for change detection strategy.

Bug, feature request, or proposal:
Bug

DatePicker not working properly for change detection strategy.
Datepicker showing only today date on view ,not value from service.
onchange value format showing wrong. I used format ('y-MM-dd'); But value Im getting is (dd-MM-y)

Which versions of Angular, MD2, OS, browsers are affected?
Angular : 4.4.6
md2: 0.0.29
os : windows 10
Browser: Chrome  61.0.3163.10(64 BIT)

same issue :(

@ChenReuven I managed to fix temporarily by hiding md-datepickers input field and replacing it with my input field used in components. It worked.
Reason behind the issue is datepicker and input field not working properly for change detection strategy.

Thanks men i will try this solution. It means that u will have 2 inputs(one that u create and one related to datepicker) that will look at the same model/formcontrolname, right?

Yes both inputs share same ngmodel variable. And hide md2-input using opacity:0 or visibility hidden . don't use ng-if

Thanks bro, appriciate it :-) i hope that perminate fix will insert soob cause it CRITICAL bug.
Suppose the best answer right now is to wrap the date picker with custom element and use your suggestion fix

Till this issue is fixed, you can use the custom npm package I created by modifying their source code.
https://www.npmjs.com/package/abhi9-md2
You should rename the folder 'abhi9-md2' to 'md2' if you have an already setup project.
Or if you want to fix it manually, you can refer this
https://github.com/Promact/md2/issues/291


[Blog publishing process] Formalise the blog publish process

Recently I wrote  a blog and tried to follow this blog publishing guide. However, it is not a process. Because of this, we have different understanding on when and how to publish a blog.
We should define a process:
step 1: submit a blog
step 2: raise a github issue
step 3: the editor team to review within x days
step 4: address feedback and publish
We need to have a clear plan on this.... low bar entry and quick turnabout. We need to use a tag blog and marketing calls need to look at these issues and address them weekly.

Expectations setting is great.
Usually, if I blog arrives, it should get a reply within 3 days, max ( week days)
Your ticket blog, I didn't see it arrive. Yet I saw its closing notification that triggered my review :)
So you see, We need to enable the reviewer to be welcomed to pull Editors to the ticket if 3 days pass. All within the Active Reviewers (only a few of us currently help) Easy.
Keeping the bar low and easy to own is very important. I always say, lets try stuff and adjust. Formalizing the review process is good for us. I am very happy we are able to do this.

Hi,
I've prepared a draft with instructions on how to submit an article on MicroProfile blog
https://docs.google.com/document/d/1BwBLs88AhkOmxF8KEuTMosyPaOxOdMsUm7VJrHmIrrE/edit#
Please review and edit as required.
On the call there was also mentioned option to create an issue template that has a checklist:

Review of Google Docs draft (optional)
Review of uploaded article
Publish article
Post the Tweet

If this is going to be done, then we can add in the instructions that the git issue should be created based on this specific template. Are we going to create it? If yes, who is going to do that?

@Emily-Jiang & @TetianaFTV,
I have reviewed the document drafted, linked it to current admins document (moved to INFRA folder) & the regular guidelines.
I also checked the 2021 active wordpress Admin/Editors as well.
A major part is missing, I pulled you via comments Tetiana as content contribution always have 2 parts, "need for editors to review content" & no need to copy/edit. However, even if the google document is not dropped or requested, the Reviewer might recommend the conversation of the WordPress blog to google document to enable that "must" review. That needs to be let to the sole discression of the MicroProfilers attending the review.
Also, This document serves to the Editor/Admins who help with publishing. So it needs to speak to them as well. 365. It is important for the Reviewer to feel welcomed to submit a marketing forum email asking for a follow up on any review ticket, ideally the conversation lands into the Mktg call & is discussed.
The document misses the your article won't be published in the MP blog + potential reasons. This happens so it needs to be stated in the document as well.
Really good start,

Become an Author is the source of truth.
PRs can modify current process : )


Set Debug Mode does not stay the configured time (10m) and goes back to 0 shortly after setting

Environment info

NooBaa Version: 5.2
Platform: Kubernetes

Actual behavior
Setting Debug Mode from the UI should keep the system in debug mode for 10m and then set it back to 0. Instead of staying for 10m, the delayed promise to set levels back to 0 jumps after 3-5seconds.
return _set_debug_level_internal(req, req.rpc_params.level)
        .then(() => {
            if (req.rpc_params.level > 0) { //If level was set, remove it after 10m
                promise_utils.delay_unblocking(config.DEBUG_MODE_PERIOD) //10m
                    .then(() => _set_debug_level_internal(req, 0));
            }
        })
        .return();

Expected behavior

Should stay for the configured length (DEBUG_MODE_PERIOD) which is current 10m

Steps to reproduce

Set Debug Mode from the UI

Works as expected locally as well as minikube


Add tab tree structure

I would love to see support for a tree tab structure, similar to that of tree-style-tab for Firefox. Preliminarily, I could see a horizontal tab tree working as follows:

the tab bar shows the current tab as well as all of its siblings, side-by-side
a key combination can be used to move up and down the tree, e.g. ctrl-up & ctrl-down
when cmd-shift-e is pressed, the current tab and its siblings stay in their positions, but an additional tab bar containing all the children of the current tab appears below. These children should update based on which tab is currently selected. Moving up in the tab tree selects the current tab's parent, and the tab bar now displays the parent and all of its siblings.

A vertical tab bar would also be useful, but is somewhat less so on normal 16:9 screens, especially considering that many sites are optimized for this aspect ratio. I'm quite busy with classes this semester, but I would love to contribute to min in some way when time allows. Maybe I'll start trying to tackle tree tabs.
I didn't want to clutter up the text of this issue, but I want to say how impressed I am with this project. I am astounded by the sheer number of commits you have made, especially considering that you are working alone. I use DuckDuckGo on all of my machines, and the tight integration between DDG and min is ridiculously awesome. This project deserves so many more stars, and I'm definitely going to tell my friends about it. Keep up the great work, and I'll definitely keep an eye on how min is progressing!

What’s the main benefit of tree-style tabs? I don’t use them myself so I’m wondering:

Are they useful as a visual thing, where you can quickly see how your tabs are related?
Or is the benefit that it makes navigating around the tabstrip using keyboard shortcuts easier?

I'm wondering if this is something that should be built into expanded tab mode, rather than the regular tabstrip. It seems like it would be difficult to keep the regular tabstrip simple, while also making it work with large trees of tabs.

I often have hundreds of tabs open, organized into separate windows by project. I find that I can't really have any more than 20 or so tabs in a window before it becomes impossible to both read the tab titles and navigate to different tabs. A vertical tab bar solves the first problem because each tab takes up much less vertical space than horizontal. A tree structure solves the second problem, because it allows me to group tabs in ways that make sense to me.
For example, one of my windows currently has around 100 tabs, but only ~20 top-level tabs. The tabs are all distinct from each other but have related tabs as children. E.g. one is the MDN reference for RegExp, and its children are several other tabs about regexes. Another is the reference for the SVG coordinate system. Its children are various other SVG references and other similar tabs.
But yeah, I think it would do best in just expanded tab mode. The regular tabstrip can just show the current tab and its siblings, as usual.

Have you tried Firefox's tab groups feature?
It sounds like multiple levels of nesting aren't important (are they?).
If that's true, Min could have something like tab groups. Expanded tab mode could show all the groups, and the tab bar would only show the tabs in the current group (like Firefox). The expanded tab mode would also have a button to create a new group, and would allow dragging tabs between groups, dragging a tab into a new group, etc. To use your example, you could have a group with all the regex-related tabs, and a group with all of the SVG-related tabs.
Benefits:

Unlike a true tree-style system, it doesn't interrupt the normal tab mode - if you never open expanded mode, tabs will work just like they currently do.
Easier to visualize than tree-style tabs.

The downside would be that there can only be one level of tab nesting.
The expanded tab mode UI could work in a couple of different ways:

Tiles for each tab (like the current expanded mode). Each tab set would take up the full width of the screen, and expand to 3-5 rows vertically before scrolling horizontally.
Tab groups would be arranged in columns, like a vertical tab list.

I've started working on this in the tasks-overlay branch.
The regular tabstrip only shows tabs for the current task. I've moved expanded tab mode into an overlay, which shows all open tasks, and also allows you to name tasks, delete tasks, etc.
I would love some feedback on this. How well does this work? What could be improved about it?
There is a binary for OS X here: https://github.com/PalmerAL/min/releases/download/v1.1.0-alpha1/Min-v1.1.0-alpha1.zip. Let me know if you would like Windows or Linux binaries to test as well.

Hey, I just downloaded the new release, it looks great! Sorry I took so long to respond, the notification email seems to have fallen through the cracks.
I used Firefox's tab groups for a while, but they removed it as of FF45, or somewhere around there. As for the tree layout, what I was thinking of wouldn't disrupt normal usage either. It would function similarly to how you have it now, except that each task could contain subtasks. Either way, only the top-level tabs of the current task would show up in the tab bar. As for whether it's actually needed, I'm going to start using Min regularly and see if I miss a true tree.
My preliminary thought is that a tree is useful for having grouped tabs inside of tasks, e.g. you could have a "SVG reference" task with several general tabs, as well as sub-trees for SVG elements and SVG styling, which each have tabs related to those specific topics. For people who don't use this feature, nothing should change in terms of usage or visualization.
I use Arch on my desktop machine, but if it's as easy "npm install && grunt" then I won't ask you to compile a binary. Again, sorry about the lateness, I'm still pretty busy but I'll definitely test it out and give some feedback.

Just had to add this: holy crap, Min is polished. I am in awe of the work you've put into it, it honestly feels like an app made by a much larger team.

@D1SC0tech What do you think? This branch includes some pretty major changes to the data model, so I'd like to decide whether to merge this or not. Its difficult to change a lot of parts of the browser without creating merge conflicts.

I'd say merge. I've been using Min for non-critical things for the past couple of weeks, and the Tasks model works really well.


wont play modpack

Describe your bug here.
psych engine 0.7.1h and others wont play modpacks. ive tried many times and it wont
Command Prompt/Terminal logs (if existing)
No response
Are you modding a build from source or with Lua?
Source
What is your build target?
Windows
Did you edit anything in this build? If so, mention or summarize your changes.
no

Did you remember to check if they're enabled?

Yes. I enable them, turn them on, and press escape. And it doesn’t restart
psych engine to load the mod
On Wed, Aug 2, 2023 at 2:03 PM Draconic-Hybrid @.***>
wrote:

Did you remember to check if they're enabled?
—
Reply to this email directly, view it on GitHub
https://github.com/ShadowMario/FNF-PsychEngine/issues/12983#issuecomment-1662811188,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BBVJSDXAVT4K54HUKFGJAATXTKQBTANCNFSM6AAAAAA3AVCC44
.
You are receiving this because you authored the thread.Message ID:
@.***>

Press 7 on the Main Menu and see if you can load the mod directories. Try loading a song from the mod in the Chart Editor.

I'll try that.
On Wed, Aug 2, 2023 at 3:43 PM Draconic-Hybrid @.***>
wrote:

Press 7 on the Main Menu and see if you can load the mod directories. Try
loading a song from the mod in the Chart Editor.
—
Reply to this email directly, view it on GitHub
https://github.com/ShadowMario/FNF-PsychEngine/issues/12983#issuecomment-1662940764,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BBVJSDUCFDY5SY4UAEOHWYDXTK3W3ANCNFSM6AAAAAA3AVCC44
.
You are receiving this because you authored the thread.Message ID:
@.***>

That's weird
On Thu, Aug 3, 2023 at 1:56 AM Lorekeeper49 @.***>
wrote:

Could be an issue with lua crossing since lua is almost completely changed
up now
—
Reply to this email directly, view it on GitHub
https://github.com/ShadowMario/FNF-PsychEngine/issues/12983#issuecomment-1663396560,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/BBVJSDRVNIYINJABNYTDFQDXTNDTDANCNFSM6AAAAAA3AVCC44
.
You are receiving this because you authored the thread.Message ID:
@.***>


Update workflow actions to specific commit SHAs

Codecov Report
All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 74.02%. Comparing base (1237475) to head (20c426e).

Additional details and impacted files
@@            Coverage Diff             @@
##             main       #9      +/-   ##
==========================================
- Coverage   74.18%   74.02%   -0.17%
==========================================
  Files           6        6
  Lines         616      616
==========================================
- Hits          457      456       -1
  Misses        116      116
- Partials       43       44       +1

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Feature] Add class table to staging tables

Is there an existing feature request for this?

[X] I have searched the existing issues

Describe the Feature
I use this dbt package along with fivetran to create a P&L report of QuickBooks data. However, in my use case, I want to be able to reference the class name, which becomes a hassle if I have to go into each individual schema (I have 18 companies). Can you please add the class table as a staging table so I have access to the unionized data?
Describe alternatives you've considered
I am in the process of creating my own staging table with the unionized data but thought I would give it a shot to ask the experts.
Are you interested in contributing this feature?

[ ] Yes.
[X] Yes, but I will need assistance and will schedule time during your office hours for guidance.
[ ] No.

Anything else?
No response

Hi @caseymkc, thanks for opening this! I'll circulate this with my team, but in the meantime I recommend forking the package and adding that staging model for class in your local repo for more immediate use. I also see you're willing to share that PR-- Let me know if you have questions about the process!

Also @caseymkc , I just wanted to clarify you want to be able to bring in the class field name specifically. Since class_id  already exists in our end models, the class table would be brought in downstream via joining on class_id and source_relation (since you're using union schema feature to grab all 18 schemas) in order to retrieve the name.

@fivetran-reneeli I'm so sorry I meant the department table.. You're right class is already a staging table.

No worries! Though to be clear class isn't a table used in our package currently, neither is department. Here is a list of our staging models.
We can look to see if it's possible to add either table in, but it would help to get clarification on how class name or department name would get used in the end profit and loss model.  Currently the grain is set at period and account. As class_id already exists in that P&L model, it would be straightforward to bring in class_name, but department I am less familiar with.

I know you mentioned you're making your own staging table, so let us know if there's a PR you'd like us to take a look at, or if you want to discuss this live!

My apologies - I did mean class haha. We got out own staging class table working.


add barbajs, frankenphp & milvus icons

Add New Icon(s)
Icon Details

Icon ID
Icon

barbajs

frankenphp

milvus

Notes

added barbajs, frankenphp & milvus icons & updated readme file
closes #226 , closes #183 , closes #243

Checklist

[x] Added icon(s) to the appropriate directory
[x] Updated documentation with the new icon(s)
[x] Verified the icon(s) display correctly
[x] If the icon has light and dark, verified the auto switch works correctly
[x] The Svg does not contain any sort of strokes

strokes in the barbajs logo, after that it should be all good

mb, it's good to go now


New helper to ease logging configuration

Coverage decreased (-0.4%) to 93.148% when pulling db3b8d6a7b153f8f34dd93f0f9f58bde8cfe6b43 on feat/logger-ext into 1e854d74aa9ee1c1733c38f6e842f4e76159e028 on master.


Validate if api calls used by scripts still work [2h]

Validate that the api calls used by other scrips (upload to playstore, submit to facebook, ...) still work and return the correct info.

Updated the scripts


Remove DB dependency from get_all_node_results

Acceptance Criteria

[ ] get_all_node_results() in result.py uses API instead of DB to generate the list of results

Will there be a separate method that queries Result.lattice.transport_graph directly?

I think that would be one way to do it? I was thinking of enhancing the API, so get_all_node_results can get the lattice and electron data it needs from the API instead of the DB. But maybe it's easier to get it from the in memory Result object?

Yes I vote for in memory syntax. The goal is for result object, once queried to contain all information presented by methods instead of making costlier http calls as at that point of post queried result space, the users work is scripting in Jupiter notebook for rapid analysis.

PS: we can backlog an issue to deal with large objects in our optimization releases.


bug:

Prerequisites

[X] I have searched for existing issues that already report this problem, without success.

Script Version
FUT AutoBuyer 1.6.8
Platform
Webapp
Current Behavior

I get this from time to time
Expected Behavior
I am not sure
Steps to Reproduce
Install tempermonkey
Install js script
Additional Information
No response

:one: DISABLE/REMOVE TAMPERMONKEY (if you have it installed!)
:two: Click on this direct link - https://chrome.google.com/webstore/detail/fut-enhancer/boffdonfioidojlcpmfnkngipappmcoh
:three: Press 'Add to Chrome'
:four: Press 'Add Extension'
:five: Once added, go to https://www.ea.com/en-gb/fifa/ultimate-team/web-app/ & the bot will appear
Here's a YouTube Tutorial on how to install it - https://www.youtube.com/watch?v=cq9hG1BBI4E