upgrade script does not handle long node name properly

Node runner script does not respect name type in vm.args and runs install_upgrade.escript which itself only works with shortnames, and finally tries to start node with name like<EMAIL_ADDRESS>and of course fails.

Hi.
This issue is still actual and can be reproduced by just following https://github.com/rebar/rebar/wiki/Upgrades up to (not including) "Deploying with release_handler" and running instead
mv dummy_0.2.tar.gz dummy_0.1/releases/
dummy_0.1/bin/upgrade dummy_0.2

The key pre-requisite is vm.args file should contain -name <name>@<host>, not -sname <name>!
So, steps are:

vm.args file should contain -name <name>@<host>, not -sname <name>
./bin/<name> start should be started using this vm.args
release upgrade should be generated by rebar generate-appups rebar generate-upgrade or in some other way
you should try to apply this upgrade using rebar-generated helper ./bin/<name> upgrade <name>_<version>

It will fall with
=INFO REPORT==== 8-Jan-2015::19:35:58 ===
Can't set short node name!
Please check your configuration
escript: exception error: no match of right hand side value
                 {error,
                     {{shutdown,
                          {failed_to_start_child,net_kernel,
                              {'EXIT',nodistribution}}},
                      {child,undefined,net_sup_dynamic,
                          {erl_distribution,start_link,
                              [['dummy@<IP_ADDRESS>_upgrader_21087',shortnames]]},
                          permanent,1000,supervisor,
                          [erl_distribution]}}}

So, issue is in https://github.com/rebar/rebar/blob/master/priv/templates/simplenode.install_upgrade.escript#L34-L52
It forces shortnames by net_kernel:start([MyNode, shortnames]) and generate improper node name<EMAIL_ADDRESS>instead of<EMAIL_ADDRESS>Suggested solution:
fix make_script_node to generate proper node name and fix start_distribution to use short or long node name depending on presence of @ in NodeName.

The thing could probably use a proper shortname if it used something like dummy_upgrader21087@localhost instead of <IP_ADDRESS>, which is a FQDN and requires long names.

@ferd I think upgrade functionality from https://github.com/rebar/rebar/blob/master/priv/templates/simplenode.install_upgrade.escript should be moved to https://github.com/rebar/rebar/blob/master/priv/templates/simplenode.nodetool , which already supports -name / -sname and other options.

@ferd I'll make a pull-request, which moves hot upgrade functionality to nodetool

@seriyps you may want to instead look at the relx templates that have gone through a lot of improvements https://github.com/erlware/relx/tree/master/priv/templates

@tsloughter yeah, interesting. I think I first just copy current functionality (to fix bug) and, next, maybe apply relx improevements of install_upgrade.escript
P.S.: relx install_upgrade.escript has the same bug, but it forces longnames, while rebar forces shortnames!

So, #426

@seriyps wrote:

next, maybe apply relx improevements of install_upgrade.escript

Good idea!


test: TestListEmptyDir expected behavior does not apply to some file hosting service

https://github.com/beyondstorage/go-storage/blob/813c8a192cfc9f43eac39d66a910cf8d6512b7ec/tests/storager_list.go#L86-L96
path is currently a non-existent path for the service. When listing a non-existent dir:

For object storage services, ListObjects will get an empty Contents in ListBucketResult, in line with the expected behavior.
For fs, although list does not return an error, we will get no such file or directory error when calling Next().
For dropbox: ListFolder will return path/not_found/... error.

Maybe TestListEmptyDir should list an existing empty dir?
How about CreateDir(path) before List? But the service should support CreateDir feature. I'm not sure it's appropriate to check s.p.store.Features().CreateDir in TestListEmptyDir. If we judge in TestList, all of test cases for List may not be executed.

After adding CreateDir(path) before List, file hosting services basically passes the test now, but the object storage services no longer passes the test:

TestListEmptyDir will get the object workdir/path/ with size 0 in s3 and cos, maybe more services.
TestListEmptyDir test case will be passed when testing service s3 with minio.


As a user, I want to see a ranking of players who have played the most games on a particular hero.

As an SK player, I'm just curious how many players have played more SK games than I have. I think it would be fun to have a ranking of number of games played, by hero.
We could add a tab to https://www.opendota.com/heroes/16, where you can select between the current "rankings by score", and an additional "rankings by games played".
Acceptance criteria:
User goes to hero page. Selects a tab "rankings by games played". Sees top 100 players by number of games played on that specific hero.

We could do this if it's quarterly like hero rankings currently are.  We'd just keep another zset that counts matches played.  The only concern is that rankings currently take up a lot of Redis space, and keeping track of a count would double it.

I haven't looked into how the current "rankings by score" is stored, but for all-time games played, the top 100 for each hero probably doesn't change that often. You can likely just store the top X number, then have a manual update button similar to the "profile refresh" or "match parse" functionality for players who want to see themselves on the leaderboards.
Separately, you could then more frequently update the "rankings by games played" for just the top X number, say daily.

The problem is that if we want to do all time, it's a pretty expensive operation to iterate over the 1.3 billion already played matches to find the counts.
If it's opt-in (the user has to do some action to show up on the leaderboard) then it's likely that many top players won't show up at all.

Do you already have the "matches played" cached somewhere? You could just iterate over the player base once, and that gives you say, the top 150 most-played rankings for all the heroes. That solves the problem of top players not showing up.

After performing the indexing once, I'm then suggesting that only the top 150 per hero need to be updated frequently. Thereafter, the opt-in is only for players who are interested in being updated more frequently as part of the leaderboards. Again, I suspect that the top 150 probably don't change too frequently because it's hard to catch up to the number of games played. This should solve the need to iterate over every single match in existence all the time.
You could then just display the top 100 for each hero.

The hero counts aren't cached at the moment, Cassandra gets read on every request (though it's faster than iterating over every match).
I think there are something like 7 million players in the database, so we'd still have to make a request for each of them.  If we request at around 10 reqs/sec we could finish the indexing in about 10 days?  There might be some performance degradation during this time.
Data inconsistency is still an issue, though.  Unless we schedule a periodic full re-indexing then players who climb the leaderboards won't ever show up (unless they opt in)

What if the service only indexes players when db selects for matches played are being done already?
This would eliminate the need to iterate through 7 million players at once, since a large number of those probably aren't logging in every day and checking their matches-played list. The full-indexing would be spread out over time, thereby reducing the immediate performance impact.
The data inconsistency would still be there in that the ranked list wouldn't be absolutely comprehensive of all players. However, you could also argue that the point is to have a ranked list of relevant players, and that players who rarely check their stats (or login) shouldn't be treated much differently from players who don't have an account. We can also take advantage of the logic that is already present for not-parsing-replays-if-player-hasn't-logged-in-within-X-days feature you have.

That might work, we could maybe add a separate call to add data to redis whenever a heroes page is requested (the API gets called on the overview load as well)
Then add another endpoint to read the data, maybe we can do that and then inspect the data to see if it looks ok before adding the table to the UI to display it.

That sounds like a good idea! Take a look at the data first, then see how much of it actually makes sense to display as a ranking.
In terms of the issues backlog, what can I do to help this become prioritized?

Fastest would be to implement it yourself (I can help you get started).
Otherwise, will need to wait until I have some free time (maybe this weekend?)

I'm a really terrible developer (so it's definitely not going to be fast), but I think this would be a good learning opportunity. Where do I start? Maybe you could help me break down what functionality I need to add to where.

You'll want to clone the odota/core project and then get it running.
The following function should be modified to make a redis call to save the count of times played for each hero: https://github.com/odota/core/blob/master/routes/spec.js#L1425

Do this right before the res.json() call which returns the data.
You'll want to use a redis zset to store the data (see the ZADD documentation: https://redis.io/commands/zadd)
You might want to trim the data as well so only the top 100/150/whatever number are stored, (https://redis.io/commands/zremrangebyrank)
After you get the data stored, add an endpoint to retrieve the data.  You can copy/paste one of the existing endpoints like '/rankings' (maybe call it mostGames) and then write the function to retrieve the data from redis (https://redis.io/commands/zrange WITHSCORES) and then return the data with res.json()
Then test your new endpoint by hitting your development API.


[Questions] Requests Always served by Same VM. NAT Gateway redundant?

I have questions about this supposed to be a simple demo article.
Question 1
The same VM always serve the requests. I have to stop it, then the other VM returns the request. Any idea why?
Question 2
By default, all outbound requests are allowed within a subnet, why is the NAT Gateway introduced given this article should be made simple to read instead of a complicated one?
This article could have been made much simpler to understand.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 5508502b-91e9-3cd5-c976-23c9b9bac1af
Version Independent ID: cf1a73b1-283f-fb88-ebf6-06d68143f5ec
Content: Quickstart: Create a public load balancer - Azure portal - Azure Load Balancer
Content Source: articles/load-balancer/quickstart-load-balancer-standard-public-portal.md
Service: load-balancer
GitHub Login: @greg-lindsay
Microsoft Alias: greglin

@PingPongSet
Thanks for your feedback! We will investigate and update as appropriate.

Hello @PingPongSet,
Base on your questions above.

The same VM always serve the requests. I have to stop it, then the other VM returns the request. Any idea why?

Azure Load Balancer uses a five-tuple hashing algorithm for the distribution of inbound flows. If you have selected session persistence to be None in the Azure portal then Hash based distribution model is used where the hash is used to route traffic to healthy backend instances within the backend pool. The algorithm provides stickiness only within a transport session. When the client starts a new session from the same source IP, the source port changes and causes the traffic to go to a different backend instance.
If you are using Session Persistence   connections from the same client will go to the same backend instance within the backend pool.

By default, all outbound requests are allowed within a subnet, why is the NAT Gateway introduced given this article should be made simple to read instead of a complicated one?

Yes, Azure does provides a default outbound access IP for VMs that either aren't assigned a public IP address but this is not a recommended method for out bound connectivity.
In this tutorial Azure NAT Gateway is used for outbound connectivity as NAT is fully managed and highly resilient and is recommended for outbound connectivity. You can go through this documentation for additional details.
Hope this resolves your query. As we primarily monitor this channel for doc enhancements/bugs we will now proceed to close this issue out. Please tag me in your response if you have any additional questions. Thank you!


Image Preview from "Resources" or "FileSystem"

I would like to submit a request to preview images when clicking on the name from the Resources Tab or the FileSystem Tab down below.
Currently, when you double-click on a PNG image under FileSystem, it opens highlights the name of the file under Resources. Maybe when you double-click on the file name under Resources it could force the Inspector Tab to show a small preview of what the image looks like. The current implementation changes the Inspector Window, but does not select the tab and make it visible. A PNG image is described by it's dimensions, compression, mipmaps, quality, etc.

this feature is planned for 1.1!
On Thu, Jan 8, 2015 at 8:45 PM, BodhiPeace<EMAIL_ADDRESS>wrote:

I would like to submit a request to preview images when clicking on the
name from the Resources Tab or the FileSystem Tab down below.
Currently, when you double-click on a PNG image under FileSystem, it opens
highlights the name of the file under Resources. Maybe when you
double-click on the file name under Resources it could force the Inspector
Tab to show a small preview of what the image looks like. The current
implementation changes the Inspector Window, but does not select the tab
and make it visible. A PNG image is described by it's dimensions,
compression, mipmaps, quality, etc.
—
Reply to this email directly or view it on GitHub
https://github.com/okamstudio/godot/issues/1154.

this feature is planned for 1.1!

And it was implemented :)


Pass ymin and ymax through formula

Creating a new issue from this comment by https://github.com/grantmcdermott/tinyplot/issues/224 by @zeileis

Would we want to be able to specify ymin and ymax through the formula?

Replies from the same thread:
@grantmcdermott

Related question: Would we want to be able to specify ymin and ymax through the formula?

Yes!
Honestly, this is something that I'd love to have a "good" syntax for. I'm all ears if you have ideas. We should probably open a separate issue to discuss.
P.S. The way that base plot handles this AFAICT is to allow ymin/ymax passing through NSE. I can't think of an example off the top of my head, though...

@vincentarelbundock

Not a strong preference, but I feel that we already have arguments called ymin and ymax, and it's generally good to have only one way of doing things, and to avoid overloading the formula. by is a very common use-case, and I think it might be confusing if the formula is used for more than that.
But again, this is a relatively weak preference.

@zeileis

Well, we can either use standard non-standard evaluation when building the model.frame(). This would be like weights = ... or offset = ... in regression functions like lm() and glm(). So we could include ymin and ymax there. BTW: weights might also be handy, especially for categorical data but also for pre-binned data.
Alternatively, we could have formulas like cbind(ymin, y, ymax) ~ x | by. But then we would need to decide whether to turn multivariate responses into matrices or data frames etc.


Add back-to-top button

Add a URL that redirects to the contents anchor at the end of a header-categorized list.
It will make it much easier to redirect to the content.
That option is available with Awesome-Go(avelino/awesome-go). Check to see if it can also be implemented here.

Good idea! Any PRs are welcome!

Thanks. Created a pr #1342 for this issue.


Update documentation

Many api functions are missing documentation

I can take care of this

Do you know which apis are missing docs ?
fyi, I've also added the command npm run build-doc that should automatically generate documentation. Maybe we could have a pre-commit hook that runs it every time before making a commit

Do you know which apis are missing docs ?
Most of the RootChain methods to do with exits don't have docs.
Some methods in ChildChain as well...

fyi, I've also added the command npm run build-doc that should automatically generate documentation. Maybe we could have a pre-commit hook that runs it every time before making a commit

Cool. pre-commit might be too often though, how about pre-push instead?


Upgrade tested version compatibilities

New Feature

[X] Code conforms to style guide
[X] Doctests
[X] Unit-tests
[ ] Documentation
[ ] New line item in releases.md
[ ] Info added to literature directory

Closing in favour of #331


Error when bot is deployed on Azure

Bot Info

Bot handle: BotAppProd
App ID: 6407709a-6db4-43d2-b9ee-ad63f174fa5b
SDK Platform: .Net
SDK Version: 3.9
Active Channels: DirectLine, Chat
Deployment Environment: Azure App Service

Issue Description
Bot works fine with the emulator, but gives 500 error code when deployed on azure. Error is same in the WebChat and DirectLine channel.
Expected Behavior
Bot works in the similar manner when deployed.
Actual Results
There was an error sending this message to your bot: HTTP status code InternalServerError
This message is displayed on the bot.

The 500 error is a generic message that basically means something is wrong within your bot code.  Is there any way you can share your code in a repo, or debug and post the minimum code to reproduce?

Thank you, the issue is resolved.


Right click behaviour

Hi, cool component, I have an issue where the right click would pop the context menu, only sometimes it will catch the rigth click.

I've tried  <div class="upload-container" on:contextmenu={handleContextMenu}>  with ```
handleContextMenu(event) {
event.preventDefault();
}

Solved modifying this ```

 e.preventDefault()} ...
``` at BoxDrawer


restore file to restore root owned file

Output of restic version
restic 0.4.0 (v0.4.0-0-ga7c2f28-dirty)
compiled with go1.7.5 on linux/amd64
Steps to reproduce the behavior
I backup by ngnix config that owned by root, restore fail with
error for b-rest/config-name: lchown b-rest/config-name: operation not permitted
Lchown
restic.Node.restoreMetadata
/tmp/restic-build-119647226/src/restic/node.go:147
I think we need option to ignore permissions when restore, since I primary backup file content not permissions.

Agreed, I'm already working on this: Split errors for restore into two categories: Fatal and non-fatal (e.g. permission errors), and add an option to ignore non-fatal errors for restore.

Any idea on how to address?
Followed the guide and created a service account called restic then set these permissions on binary at '/usr/bin/restic'

0750
setcap cap_dac_read_search=+ep

-rwxr-x---. 1 root restic 12M Jun 14 11:08 /usr/bin/restic

Please see my solution in #1766.

This issue has already been solved in the latest version, restic will print the errors but continue anyway. I'm closing it here. Thanks!


Adds Better way of checking if corner is visible

Using snippets from jQuery UI and doing lots of math, we can check if the corner is visible to the user or not, for scrolling usecases like this: https://i.imgur.com/QAv57a4.gifv
Previously, the lower items would be considered out of bounds, and would not have the corner when selected.
User needs to run  updateCornerPosition themselves within the scrolling parent, however:
$('.scrolling-wrapper').scroll(function(){
    $(this).jexcel('updateCornerPosition')
})

all those spaces were cleaned by a lint. the real code starts at line 2503 and at the very bottom for the helper functions

This has been improved on the new version


[BUG] FE CPU 100%

Describe the bug
when i use stream load , cpu will increate to 100%
To Reproduce

start fe and use stream load
cpu will increase to 100%
stop fe
cpu will recover to normal

Expected behavior
cpu should't cost so much cpu
Screenshots

Additional context
1 FE 1BE

what's your doris version?

I use jstack, i found it hang here

@caiconghui master
I found it was a BUG when no BE avaliable

is there BE alive when CPU 100% ?


feat: just backport-pr

Using https://github.com/sqren/backport
Requires creating an access token and putting it in a local configuration file.
I think it uses the same code our backport bot is using, but will execute it locally using npx.

Certainly looks fancy, let's see how useful it actually is.


Updated Cribl logo

Summary
Updated Cribl logo and changed the config.yml file to reflect the new file.

Sorry @nr-jbiggley but the repository name was recently changed and it seems that is giving merge issues with your PR. Your changes are referring to the packs/ folder while they should be in the quickstarts/ folder.
Might be easiest to just create a new fork to make sure you have the latest version, let me know if you need some help

Resubmitted PR


Create Personas and Credible User Stories

Look up different UX design related projects/case studies similar to TileBot (educational board games, applications/resources that teacher-student use in school settings, etc.)
Questions to think about: Who exactly needs help ? Teachers/students? Age groups? Grade level? Demographic ? Challenges in the academic setting ? What is the main concept behind our design/application?
Group Checklist

[ ]  Find out all about users/target audience: Children of different age-grade level groups, teachers with different teaching styles/experience, ## add more ##

[ ]  Make an affinity diagram : goals+motivations+behavior+attitude

[ ]  Minimize Performance Load: Cognitive vs Physical Workload

[ ] Match mental models of user stories

[ ] Affordances vs False Affordances

[ ] Constraints

[ ]  Combine user experiences to map/create personas

[ ]  Update Project Report

Individual Checklist

[ ]  Cesar Credible User Story

[ ]  Cesar Persona

[ ]  Juan Credible User Story

[ ]   Juan Persona

[ ]  Carlos Credible User Story

[ ]  Carlos Persona

Teacher Themes:

[ ]  Main Themes:  "creativity, improvisation, planning, organization"
[ ]  Revolving Themes: Developing pedagogical content knowledge, use existing knowledge to create new knowledge, create instructions, develop questions
[ ] Text Based Interface ?
Research Based : Game designed for 2 people builds pair programming. Adding more people into gameplay can add confusion, longer wait periods, etc.


is the matchId definitely an INT32

is the matchId definitely an INT32?
is there a 32-bit matchId and a 64-bit matchId  both?
64-bit matchId = 32-bit matchId +<PHONE_NUMBER>0265728 ?
thanks!

Hello. I didn't hear about 64-bit matchid.
Maybe someone can answer more (check on dev.dota2.com)

Hello. In your code, matchId is regarded as int.
However, matchId is an unsigned int . What if matchId is greater than 2^31-1?
Line 24 in MatchMapper.php:
$this->_match_id = (int)$matchId;

For current moment match_id is about 2^30, so i have some time to think about this.


Duplicate implementation of (multi) controlled gates

What is the expected enhancement?
Qiskit/qiskit-terra#3600 implemented the Cry gate in Terra, therefore we can remove it from Aqua. This duplication might also cause some bugs, since the Aqua version is imported after Terra's and we override the definition as proper CryGate with ours.

Yes, there are a number of gates that are in the process of being moved (or have been to Terra). These all need to be taken care of (PS. The CRy rewrite was submitted here first as a PR https://github.com/Qiskit/qiskit-aqua/pull/745 which I suggested was re-opened against Terra which has been done). I asked that when the move was complete to Terra a PR be opened here to delete them (yes at present they are duplicated - except that the CRy was rewritten, as you have seen, to subclass Gate and the one here not)

Yes, the multi control gates are already in Terra, thanks for the hint. So these should also be removed from Aqua, and the tests moved to Terra since they don't have them yet.
@woodsp-ibm do you know why the logical gates haven't been moved? They seem to be the only ones missing.

I am not aware of the current status, last time I had checked with Kevin work was on hold at that time due to an imminent release being prepared.

Terra released 0.11.0 last week, so this should be good to move forward.

Once Qiskit/qiskit-terra#3714 is merged, we can remove the gates from Aqua (except the boolean logic one).

Gates left in Aqua are mcmt and boolean logic. These were never duplicated and the duplicate gates, the subject of this issue have been removed. Should this be closed and if needed another issue to manage mcmt and/or boolean logic gates if change there is still expected?


server.crontab() not identifying cronjobs

Describe the bug
If this is not a bug I am horribly confused on how this is actually supposed to work. 🙃
If an unidentified cronjob exists in your crontab and you attempt to identify this cronjob with the cron_name argument, this does not add an identifier to the cronjob. You must then make another call to server.crontab() with the same cron_name if you want to make changes to the cronjob. This time however, instead of overwriting the existing cronjob that you just tried to identify, server.crontab() will create a new entry in your crontab - appending it to the end of the file - leaving you with the cronjob you initially tried to identify as well as the new cronjob with the identifier you tried to apply to the initial cronjob.
To Reproduce
Have a crontab with some cronjobs in it that aren't identified by PyInfra, then run server.crontab() against your crontab in an attempt to identify the cronjobs, and then modify them afterwards.

Operation code & usage

server.crontab(
    name="Identify Cron",
    command=f"/usr/bin/php{php_version} {old_estore_dir}/bin/magento cron:run 2>&1 \
| grep -v \"Ran jobs by schedule\" >> {old_estore_dir}/var/log/magento.cron.log",
    minute="*",
    hour="*",
    month="*",
    day_of_week="*",
    day_of_month="*",
    present=True,
    user=f"{username}",
    cron_name=f"{site}_cron",
    _sudo=True,
)

server.crontab(
    name="Modify Cron",
    command=f"/usr/bin/php{php_version} {estore_dir}/bin/magento cron:run 2>&1 \
| grep -v \"Ran jobs by schedule\" >> {estore_dir}/var/log/magento.cron.log",
    minute="*",
    hour="*",
    month="*",
    day_of_week="*",
    day_of_month="*",
    present=True,
    user=f"{username}",
    cron_name=f"{site}_cron",
    _sudo=True,
)

If I attempt to identify and modify either of the cronjobs, I get a brand new cronjob appended to the end of the file as shown, with the cron identifier. In this example I tried to edit the staging_cron
Before
* * * * * /usr/bin/php8.2 /opt/<name>/apps/prod/estore.e483eac1/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/prod/estore.e483eac1/var/log/magento.cron.log

* * * * * /usr/bin/php8.2 /opt/<name>/apps/staging/estore.1312e8dd/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/staging/estore.1312e8dd/var/log/magento.cron.log

After
* * * * * /usr/bin/php8.2 /opt/<name>/apps/prod/estore.e483eac1/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/prod/estore.e483eac1/var/log/magento.cron.log

* * * * * /usr/bin/php8.2 /opt/<name>/apps/staging/estore.1312e8dd/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/staging/estore.1312e8dd/var/log/magento.cron.log

# pyinfra-name=staging_cron
* * * * * /usr/bin/php8.2 /opt/<name>/apps/staging/estore.656bc6af/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/staging/estore.656bc6af/var/log/magento.cron.log

Expected behavior
The cronjob should be identified with the first call to server.crontab(), prepending the cron_name identifier/comment on the line above the cronjob. Then, the second call to server.crontab() should overwrite this cronjob with the new contents if the same cron_name is used, not appending a new cronjob to the end of the file, and not leaving the initial cronjob behind.
 Pip

Debug info:

--> Starting operation: Identify Cron
    [pyinfra.api.operations] Starting operation {'Identify Cron'} on @local
    [pyinfra.api.facts] Getting fact: server.Crontab (user=<name>) (ensure_hosts: None)

[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c '! command -v crontab >/dev/null || crontab -l -u <name> || true'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
    [pyinfra.api.facts] [@local] Loaded fact server.Crontab (user=<name>)
    [pyinfra.api.host] [@local] noop: crontab /usr/bin/php8.2 /opt/<name>/apps/staging/estore.1312e8dd/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/staging/estore.1312e8dd/var/log/magento.cron.log exists
    [@local] No changes

--> Starting operation: Modify Cron
    [pyinfra.api.operations] Starting operation {'Modify Cron'} on @local
    [pyinfra.api.facts] Getting fact: server.Crontab (user=<name>) (ensure_hosts: None)

[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c '! command -v crontab >/dev/null || crontab -l -u <name> || true'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
    [pyinfra.api.facts] [@local] Loaded fact server.Crontab (user=<name>)

[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c 'crontab -l -u <name>> /tmp/pyinfra-99800753387583ff6590a828b5571df23d46c9b6'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c 'echo '"'"''"'"' >> /tmp/pyinfra-99800753387583ff6590a828b5571df23d46c9b6'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c 'echo '"'"'# pyinfra-name=staging_cron'"'"' >> /tmp/pyinfra-99800753387583ff6590a828b5571df23d46c9b6'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c 'echo '"'"'* * * * * /usr/bin/php8.2 /opt/<name>/apps/staging/estore.656bc6af/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/staging/estore.656bc6af/var/log/magento.cron.log'"'"' >> /tmp/pyinfra-99800753387583ff6590a828b5571df23d46c9b6'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
[pyinfra.connectors.local] --> Running command on localhost: sudo -H -n sh -c 'crontab -u <name> /tmp/pyinfra-99800753387583ff6590a828b5571df23d46c9b6'
[pyinfra.connectors.util] --> Waiting for exit status...
[pyinfra.connectors.util] --> Command exit status: 0
    [@local] Success

This might be relevant to me soon.
If it's an unidentified cronjob, how is it supposed to be identified when you give it a name?
I suppose if you can identify it with other means, either by line number or command content(with a flag or something) or whatsoever, it's gonna add a name to the existing item.
Sounds like a feature request rather than a bug

Yeah I might be misunderstanding what this operation capable of. I was under the impression that you were able to identify existing, unidentified cronjobs, but that doesn't seem to be the case - in which case I suppose this would be a feature request.
I've been verifying existing cronjobs using a small regex function I created. Basically it runs a regex pattern against host.get_fact(Crontab) for my specific command and returns the match if it exists. I can then pass that match in to my server.crontab(command) argument as a variable.
My thought process on how to identify an unidentified cronjob was to theoretically do something like the function I just described above, then if the cron_name argument was passed in and didn't already exist on that cronjob, it would prepend it to the cronjob, like it already does with a newly created one. Theoretically, it could delete the cronjob and replace it with the new, identified cronjob as long as it replaced in-line, not appending it to the end of the file which is the current behavior if you delete the unidentified cronjob (present=False) and replace it with the new identified one.
For the time being, I'm able to use python-crontab to accomplish what I'm trying to do here along with the regex function slightly modified from I described earlier.
def extract_active_cronjob(crontab, regex_pattern):
    regex = re.compile(regex_pattern)
    matches = []

    for cronjob in crontab:
        if isinstance(cronjob.command, str):  # Ensure command is a string
            match = regex.search(cronjob.command)
            if match:
                matches.append(match.group(0))
    return matches

# Initialize the user's crontab
user_crontab = CronTab(user=username)

# {site} is either "staging" or "prod"
pattern = rf'/opt/<name>/apps/{site}/estore\.[a-zA-Z0-9]+'

active_instance = extract_active_cronjob(user_crontab, pattern)

# If there is no active instance, create a new cronjob
if not active_instance:
    user_cronjob = user_crontab.new(
        command=f"/usr/bin/php{php_version} {estore_dir}/bin/magento cron:run 2>&1 | grep -v \"Ran jobs by schedule\" >> {estore_dir}/var/log/magento.cron.log"
    )
    user_cronjob.minute.every(1)
    user_crontab.write()
else:
    # Update the existing cronjob
    old_estore_dir = active_instance[0]
    for user_cronjob in user_crontab:
        if user_cronjob.command == f"/usr/bin/php{php_version} {old_estore_dir}/bin/magento cron:run 2>&1 | grep -v \"Ran jobs by schedule\" >> {old_estore_dir}/var/log/magento.cron.log":
            user_cronjob.set_command(
                f"/usr/bin/php{php_version} {estore_dir}/bin/magento cron:run 2>&1 | grep -v \"Ran jobs by schedule\" >> {estore_dir}/var/log/magento.cron.log"
            )

    user_crontab.write()

thanks for pointing out python-crontab. combined with other issues, I'm motivated to use python-crontab to replace current implementation, and solve issues once-for-all

Yeah I might be misunderstanding what this operation capable of. I was under the impression that you were able to identify existing, unidentified cronjobs, but that doesn't seem to be the case - in which case I suppose this would be a feature request.

I read the implementation, the crontab fact is able to locate by unique commands, but the crontab operation can't handle it. so this might also be a bug

thanks for pointing out python-crontab. combined with other issues, I'm motivated to use python-crontab to replace current implementation, and solve issues once-for-all

Yeah, no problem. So far it seems to work great and requires very little code to accomplish the task at hand. I ended up wrapping that chunk of code after extract_active_cronjob() into its own function and then used PyInfra's python.call() to call the function. Works like a charm! 🙂
Also, I was able to add comments to existing, unidentified cronjobs in the way that I wanted to initially, even though it's not necessary now. The final code looks like this:
def extract_active_cronjob(crontab, regex_pattern):
    regex = re.compile(regex_pattern)
    matches = []

    for cronjob in crontab:
        if isinstance(cronjob.command, str):  # Ensure command is a string
            match = regex.search(cronjob.command)
            if match:
                matches.append((match.group(0), cronjob))
    return matches

def modify_crontab():
    # Initialize the user's crontab
    user_crontab = CronTab(user=username)

    pattern = rf'/usr/bin/php[0-9.]+ /opt/<name>/apps/{site}/estore\.[a-zA-Z0-9]+/bin/magento cron:run 2>&1\
     | grep -v "Ran jobs by schedule" >> /opt/<name>/apps/{site}/estore\.[a-zA-Z0-9]+/var/log/magento.cron.log'

    active_instance = extract_active_cronjob(user_crontab, pattern)

    # If there is no active instance, create a new cronjob
    cronjob_modified = False

    if not active_instance:
        user_cronjob = user_crontab.new(
            comment=f"{site.upper()}",
            pre_comment=True,
            command=f"/usr/bin/php{php_version} {estore_dir}/bin/magento cron:run 2>&1"
                    f" | grep -v \"Ran jobs by schedule\" >> {estore_dir}/var/log/magento.cron.log"
        )
        user_cronjob.minute.every(1)
        user_crontab.write()
        cronjob_modified = True
    else:
        # Update the existing cronjob
        old_estore_dir, cronjob = active_instance[0]
        for user_cronjob in user_crontab:
            if user_cronjob == cronjob:
                user_cronjob.set_command(
                    f"/usr/bin/php{php_version} {estore_dir}/bin/magento cron:run 2>&1"
                    f" | grep -v \"Ran jobs by schedule\" >> {estore_dir}/var/log/magento.cron.log"
                )
                if user_cronjob.comment != f"{site.upper()}":
                    user_cronjob.set_comment(f"{site.upper()}", pre_comment=True)
                    cronjob_modified = True
                break

        user_crontab.write()

    if not cronjob_modified:
        raise Exception("Failed to create or update the cronjob.")

python.call(
    name="Modify crontab",
    function=modify_crontab,
)

I'm not 100% sure what's the cause, but the sed-based line editing seems suspicious to me
sed can't add comments(name) to the existing command
I'd prefer editing everything on controller side and send over.

Hi @kwilt. I didn't spot this issue in my PR. Can you check if my PR has fixed your issue?


Toml LSP completion takes precendence over crates

Current behavior

When getting completion for the version field, I am instead getting generic TOML LSP completion.
Expected behavior

crates.nvim takes precedence when in a Cargo.toml buffer.
Additional context/Screenshots

Using the taplo LSP for TOML.
Tested on both crates 0.3 and 0.4 recently.
Possible solution

Neovim version

nvim --version
output of nvim --version

0.9.2

nvim-cmp, with crates as one of the sources below nvim_lsp

As I said, it's up to you to configure:
https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/doc/cmp.txt#L582C45-L582C45

I can't find any way to set the priority on the source side, maybe you can look into this:
https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/doc/cmp.txt#L620C46-L620C46

I'm starting to suspect it's not a precedence issue, but that crates completion source has stopped working for me altogether. If it helps I could reword this issue.

Yeah, either that or open a new one.


[FSDE-16823] Improve printing methods for child and grand child rows in data table

https://ctl.atlassian.net/browse/FSDE-16823

The CI pipeline run successfully in https://jenkinsprod.corp.intranet:8443/job/UX-CHI/job/Productive/job/Chi/job/PR-354/1/. :white_check_mark:

You can check this PRs instance in https://c673d4e3b7f9272cc83fb4b73f4852fb080060ab-chi-pr.enterprise.ctl.io (external) and in https://nginx-pr-354-ux-chi.kubeodc-test.corp.intranet/ (internal)

The CI pipeline run successfully in https://jenkinsprod.corp.intranet:8443/job/UX-CHI/job/Productive/job/Chi/job/PR-354/3/. :white_check_mark:

You can check this PRs instance in https://8776c75fed0266a0abbc170e5bc49ee574d4940e-chi-pr.enterprise.ctl.io (external) and in https://nginx-pr-354-ux-chi.kubeodc-test.corp.intranet/ (internal)


Add commandline output for licence

According to GPL:
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
{project}  Copyright (C) {year}  {fullname}
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.

The hypothetical commands show w' and show c' should show the appropriate
parts of the General Public License.  Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
We can add a field to main.py that contains any needed licence text and options for displaying the licence

This actually may not be necessary as the docker files will not contain any code that is GPL licensed themselves. We can keep the repository  MIT licensed according to the following from the Linux Foundation
The license of dockerfiles vs. software inside containers
The Dockerfile files can be licensed under an open source license themselves. It is vital to realize that the scope of this license statement is only the Dockerfile and not the container image.

For example, the Dockerfile itself can be licensed under the MIT license but describing the installation of GPL licensed software. In a typical use case, the license of the Dockerfile and the license of the described software are entirely independent.

I think this is mainly an issue for the individual containers and their command-line interfaces. The template and documentation can simply state that command-line programs should state license terms as necessary.
I don't think this applies to our current containers

BioNetGen: MIT
COPASI: Artistic-2.0
tellurium: Apache-2.0
VCell: MIT


Strange issue, explanation needed.

Hello,
I am experimenting with example code form readme.
Arbitration service
const cote = require('cote');

const responder = new cote.Responder({ name: 'arbitration API' });
const publisher = new cote.Publisher({ name: 'arbitration publisher' });

const rates = {};

responder.on('update rate', (req, cb) => {
  rates[req.currencies] = req.rate;
  cb('OK!');

  publisher.publish('update rate', req);
  console.log('update rate');
});

Conversion service
const cote = require('cote');

const responder = new cote.Responder({ name: 'currency conversion responder' });
const subscriber = new cote.Subscriber({ name: 'arbitration subscriber' });

const rates = { usd_eur: 0.91, eur_usd: 1.10 };

subscriber.on('update rate', (update) => {
  rates[update.currencies] = update.rate;
  console.log('rates updated');
});

responder.on('convert', (req, cb) => {
  console.log('convert request ' + req);
  cb(req.amount * rates[`${req.from}_${req.to}`]);
});

Client
const cote = require('cote');
const requester = new cote.Requester({ name: 'currency conversion requester' });

const request = { type: 'convert', from: 'usd', to: 'eur', amount: 100 };

  //callback example
requester.send(request, (res) => {
  console.log(res + ' callback res');
});

  //promise example like on readme
const makeRequest = req => requester.send(req);
  makeRequest(request).then(res => {
    console.log(res + ' promise res1');
  }).catch(e => {
    console.log(e + ' promise error1');
  });

  //promise directly on send
requester.send(request).then(res => {
  console.log(res + ' promise res2');
}).catch(e => {
  console.log(e + ' promise error2');
});

So here are my questions:

Why does, when using promises, response always comes as rejected?
first I run conversion service in one tab in terminal (max osx) node conversion-service.js
then I run client.js in another tab

output:
currency conversion requester > service.online arbitration subscriber#a1fce7ec-2e47-40f3-855a-e63a98f20735
currency conversion requester > service.online currency conversion responder#52bb841b-5131-4c3c-b295-a7c99ceb4424 on 8000
91 callback res
91 promise error1
91 promise error2

Why is there, sometimes when I start client.js, no response from conversion service?
So first an intro. When there is only conversion service and client, no matter how many times do I restart the client.js, I always get the same output like above.
The problem comes when I start arbitration service alongside conversion service. (node arbitration-service.js in another tab in terminal`.

Here is long output of me trying to restart client.js few times.
matejs-MacBook-Pro:cote matejsimunic$ node client.js

Hello!       I'm arbitration publisher#f731a8e6-ab45-41b0-9c1c-c65f5e0b8391 on 8001
========================

Hello!

Hello!
Generally, you shouldn't be mixing promises and callbacks. If you are using promises and callbacks; the callbacks should be called with two arguments, first one the error, second one the response. This is the only way to make them work together. In your case, the response is the first argument, that's why you get it as error. If you do cb(null, req.amount * rates[${req.from}_${req.to}]);, it will work (I just added null as the first argument as that's the error).

The second question is documented in keys. By default cote expects every responder to respond to every type of message. Right now you have 'update rate' and 'convert' messages in the system, but a responder only responds to one of this. Since there's no segregation/partition defined, they continue to get requests in the other type as well.
You need to define keys in the responders, like in the arbitration service, put key: 'arbitration' in the responder config. And in the client, you will need to create one requester per key, so you will have an arbitration requester and a conversion requester — the arbitration requester will have key: 'arbitration' in its config and the conversion requester will have key: 'conversion' in its config.

Thank you, everything works out as described.
I would like to add async/await example to readme and maybe mention this more clear about keys in this arbitration example if that is ok?
const response = await requester.send(request);

I'd be delighted!


Typescript is crashing a lot

Type: Bug
We have a monorepo with multiple apps but typescript is crashing a lot, to fix this we have to reload the whole ts server or the project itself. Help!
 It looks like you may be using an old version of VS Code, the latest stable release is 1.85.2. Please try upgrading to the latest version and checking whether this issue remains.
Happy Coding!

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.
Happy Coding!


MapView Scale changes automatically when switching between screens

I have set the map scale to some value after loading the layers. But when I open some activity and return back to map screen, most of the time it automatically scale the map to its default value and the whole map is visible. I have tried using mapView.pause() and mapView.resume() method but didn't work. Can anyone help me out what could be the possible solution for this issue?

This is to do with the Android life cycle destroying and recreating views. You can read more about how it relates to the ArcGIS runtime here: http://developersdev.arcgis.com/android/latest/guide/handle-device-configuration-changes.htm

To your particular comment. Are you setting your viewpoint in onCreate()? I suspect what you're seeing is that some of the time your app is staying within the onResume()/onPause() parts of the life cycle and you're seeing the map's viewpoint change whenever the Android OS chooses to call onCreate().

I do implement the viewpoint code in the onCreate(), but I'm not implementing any code in the onResume() and onPause() methods apart from mapView.pause() and mapView.resume(). So I'm not sure if this is a problem with the life cycle of activities. I also tried opening an empty activity on button click and presses back button. It still shows the showing the same issue. I also tried by checking the Mapscale value in onResume() and it gives the exact value which was set previously but as soon the map shows properly it automatically change its scale to default value.

Can you share your code (or at least your life cycle code)? I think I'll need to repro this to be able to help. Thanks!

Hi TADraeseke,
I have created a sample application which demonstrate this issue.
https://github.com/minakshibh/SampleMap
Do let me know if there is anything you need from my side.

Thanks ok! I was able to reproduce this now.
What I'd recommend is instead of the drawStatusChanged listener, etc, instead in your onCreate use:
map.setInitialViewpoint(viewpoint);

map.setInitialViewpoint(viewpoint) works even after the setMap() method call.
Thanks for the help.


Remove PPT example code

Bug report
I believe we can do it with copier update
Before submitting
Please check the following:

[ ] I have described the situation in which the bug arose, including what code was executed, information about my environment, and any applicable data others will need to reproduce the problem.
[ ] I have included available evidence of the unexpected behavior (including error messages, screenshots, and/or plots) as well as a descriprion of what I expected instead.
[ ] If I have a solution in mind, I have provided an explanation and/or pseudocode and/or task list.

Fixed by #76


Merge arrays without duplicates

Thanks @khezen! Do you think you could publish to NPM when you have a chance?


Request: Add automation like the jlesage/docker-Handbreak image

Is it possible to replicate some of the automation functionality that was added to jlesage/docker-Handbreak image?
Specifically the following:

A watch folder option for MKVToolNix, so files in sub-folders are automatically added into the Job Queue;

For the above the first file in any sub-folder within the Watch Folder is added, the remaining files in the sub-folder are appended. This can be determined on alphanumeric sorting. So 'Scene 1.mp4' is added, 'Scene 2.mp4' is appended, 'Scene 3.mp4' is appended, 'Scene 4.mp4' is appended etc;

The output file from MKVToolNix follows a naming convention based on the sub-folder that contains the files to be added to MKVToolNix. So if the files that are detected in a sub-folder called 'A Movie 253', the output file from MKVToolNix will be called 'A Movie 253.mkv'

The function for monitoring the Watch Folder can be set to wait for a predetermined time after the content within the sub-folder has last changed. This will prevent the MKVToolNix application from creating a new Job before any sub-folder has all the files.

MKVToolNix can be used in so many use cases.  The scenario you are describing is one of them.  I'm not sure implementing a watch folder targeting a single scenario is the way to go...

I concur that watch folder will be very useful to have number of your images: makemkv, handbreak and this one as a single pipeline.
I don't thing it should be super complicated, but rather simple watch + hooks like what you have in handbreak image already, then each can add their own hooks with custom logic.
It will add whole new level of usability to this image, not just GUI, but command-line/batch processing.


SolutionFinder::findSolutions(...) returns multiple models for the same associated State

This problem leads to failing of Issue59Test::testIssue59WithAdditionalInvariant()
It would be a good idea to extract States from Models before entering the add-to-queue-if-not-visited-loop

see ModelChecker.java#L69

Problem originates from 41a80cb10929077db32359b1bc26d5d21ea6c9ed because invariant now stays inside of solver all the time. Debugging Issue59Test::testIssue59WithAdditionalInvariant(), it becomes clear that ModelChecker does a lot of redundant work because multiple models are generated for the same state.

The invariant should not be asserted when searching for a successor state. Maybe we should have different solver objects for invariant checking an computation of successors?

Maybe even an instance of solver for each operation? Then we could only assert the current state and nothing else has to be reloaded?

b0abec260cd7309f9e8d374aed404cb80ca53df4 introduces separate solvers / finders for invariant checking and next state computation

see also bf192209bd5565cfcbe05b96a8301ea2d4a89ddd, due to quantified variables, some states were discovered multiple times


SFTP Initial setup, then send basic file from Azure blob storage to ReportStream

Story
As a PHL, in order for my results to be sent to RS, I need to send an ORU result from an SFTP site.
Acceptance Criteria

[ ] A file can be read from Azure blob storage
[ ] A file's contents can be sent to RS
[ ] Uses the flexion org as sender (for now)
[ ] Quality scanning by sonar cloud
[ ] Automated security vulnerability updates and resolve all non-low findings

Non - Acceptance Criteria

[X] Doesn't need to be automatically triggers
[X] Doesn't need to handle zip files
[X] Doesn't need to handle password protected files

Tasks
Research

[ ] Research work needed to complete the story
[ ] Foundational: High-level research that will support this and future efforts

Engineering

[ ] Github Setup

[x] Protect Main branch
[x] Link Repo to zenhub board
[x] Require Approvals on PRs
[x] Add .gitignore file
[ ] Add code quality and security stuff to pipeline (in future we may want API testing, but we don't have APIs yet)

[x] codeql
[ ] sonarcloud (Neil working on this - was waiting on access to CDC org)
[x] SCA (software composition analysis) - we use Snyk in TI

[x] Compile app upon merge into main
[x] Add deploy step for binary to pipeline (deploying to PR when terraform changes, deploying to staging on merge to main. Deploys available for dev and internal)
[x] Require commit signing

[ ] Repo Setup

[x] Add Dependabot
[x] Add issue templates
[x] Add updated  CI/CD files
[x] ADR for Using GoLang
[x] ADR for testify framework
[x] Copy skeleton to README.md
[x] Tidy up the default Readme files
[x] Pre-commit hooks (linting, secret scanning, etc)
[x] Enable SFTP for Azure Storage account
[ ] Terraform Setup: branch terraform

[x] Reach out to CDC to get a new vnet - @halprin
[x] Swap temporary Vnet for official Vnet in Terraform.
[x] Create blob storage in all envs via terraform
[x] App Service

[x] Adjust tier for app service

[x] Additional vnet
[x] App service placed inside of private vnet
[x] Deny All public traffic
[x] Logs
[x] dev
[x] internal
[x] staging
(Not adding prod environment for now, we'll do it when we're ready)

[x] Development

[x] Code to read a file and send to report stream
[x] Environment variable setup
[x] Write generic interfaces for Azure bucket and ReportStream sending.
[x] Make Azure-specific implementation that adheres to the above interface.
[x] Make a ReportStream-specific implementation that adheres to the above interface.
[x] Docker Compose for setting up Azurite with volume (so we can get at the files locally).
[x] Have docker-compose set up the container and a test file
[x] Auth with RS using JWT/token/login endpoint

[x] Generate JWT
[x] Extract JWT
[x] Call Auth endpoint with JWT
[x] Extract Bearer token for sending and call RS

[x] Get credentials from Azure Key Vault (needs TF setup) for existing Flexion fake public health lab
[X] Need to update vault name in key.tf to be 4 characters shorter so it works in internal env.
[x] Will need to update staging key value again after creating new vault in staging TF

[x] Clean up all to-do(s) by doing them or creating new stories

Definition of Done

[x] Documentation tasks completed

[x] Documentation and diagrams created or updated

[x] Implementation guide (/ig folder)
[x] ADRs (/adr folder)
[x] Main README.md
[x] Other READMEs in the repo
[x] If applicable, update the ReportStream Setup section in README.md

[x] Threat model updated
[x] API documentation updated

[x] Code quality tasks completed

[x] Code refactored for clarity and no design/technical debt
[x] Adhere to separation of concerns; code is not tightly coupled, especially to 3rd party dependencies
[x] Code is reviewed or developed by pair; 1 approval is needed but consider requiring an outside-the-pair reviewer
[x] Code quality checks passed

[x] Security & Privacy tasks completed

[x] Security & privacy gates passed

[x] Testing tasks completed

Load tests passed
Unit test coverage of our code >= 90%

[ ] Build & Deploy tasks completed

[x] Build process updated
API(s) are versioned
[x] Feature toggles created and/or deleted.  Document the feature toggle
[ ] Source code is merged to the main branch

Research Questions

Optional: Any initial questions for research

Decisions

Optional: Any decisions we've made while working on this story

Notes

Have a service, do TF setup (need blob storage, maybe new resource group, app service or Azure functions, private vnet, ci/cd pipelines)(may need db in future, but will wait until that's more clearly defined)
read from Azure blob storage
Log into RS (using a shared organization/sender, and using a shared key - can probably use Flexion sender/receiver for initial testing).
Send file to existing RS endpoint

Azure help desk ticket REQ0302028 created to track the creation of three new VNETs in our three Azure environments.

Blocked on sonar cloud, and Vnet.

I've got a meeting on the books with Jeremy Page, who is an admin of the SonarCloud organization, to get us unblocked.


Is it possible to connect with Tensorflow?

Hello.
I want to implementing some machine learning algorithm to control my parrot mambo, it's possible to connect with tensorflow library?
Thanks

Example of doing so here:
https://github.com/TanerDavis/Gesture-Images-For-Training


get undefined use express with readline

         // my express server code
	 const fs = require('fs');
	 const readline = require('readline');

	 async getLog(req, res) {
	  const _this = this;
	  try {
		const { createReadStream } = fs;
		const log = '/opt/example.log';
		const readStream = createReadStream(log);
		const rl = readline.createInterface(readStream, res);
		rl.on("close", () => {
		  console.log("end");
		  readStream.close();
		  res.end();
		});
	  } catch (e) {
		console.error(e);
		res.json({
                   code:2,
                });
	  }
	}

   // my browser code
   fetch(getLog)
    .then(async response => {
      const reader = response.body.getReader();
      do {
        let { done, value } = await reader.read();
        console.log(value);   //I got undefined here
        if (done) {
          break;
        }
      } while (true);
    })
    .catch(e => {
      console.error(e);
    });

Hey @assmdx, I am not sure you have an issue with express here.  Nothing you are doing in this example uses express, so I would go ahead and try to do this same with the base http package from core.  My guess is that it is something to do with the readline package, but I have not used it.  If you find that you cannot reproduce this error with the core http package and again think it is an express issue, please write a more complete example and repopen.


melonDS: 0.9 -> 0.9.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/metapackage/melonDS/versions.
meta.description for melonDS is: "Work in progress Nintendo DS emulator"
meta.homepage for melonDS is: "http://melonds.kuribo64.net/"
meta.changelog for melonDS is: ""
Updates performed

Version update

To inspect upstream changes

Release on GitHub

Compare changes on GitHub

Impact

Checks done (click to expand)

built on NixOS

The tests defined in passthru.tests, if any, passed

0 of 0 passed binary check by having a zero exit code.

0 of 0 passed binary check by having the new version present in output.

found 0.9.1 with grep in /nix/store/kdy4kbxfi8m69cb4xh98h91fbj55g5am-melonDS-0.9.1

directory tree listing: https://gist.github.com/7dfe3b3f7139b6cb6542748965fcd475

du listing: https://gist.github.com/ec6a6a0f53481a4b7e78a3e88612574d

Rebuild report (if merged into master) (click to expand)

4063 total rebuild path(s)

1329 package rebuild(s)

1200 x86_64-linux rebuild(s)
1105 i686-linux rebuild(s)
579 x86_64-darwin rebuild(s)
1179 aarch64-linux rebuild(s)

First fifty rebuilds by attrpath
DisnixWebService
R
abcl
adapta-gtk-theme
adoptopenjdk-icedtea-web
alloy
alloy4
alloy5
ansible-lint
antlr
antlr3
antlr3_4
antlr3_5
antlr4
antlr4_8
apache-airflow
apache-jena
apache-jena-fuseki
apacheKafka
apacheKafka_2_4
apacheKafka_2_5
ape
apeClex
apktool
appdaemon
apprise
archivy
areca
asciidoc-full
asciidoc-full-with-plugins
autofs5
av-98
avian
aws-google-auth
axoloti
azure-cli
azureus
backintime
backintime-common
backintime-qt
bareos
bazel
bazel-remote
bazel-watcher
bazel_0
bazel_0_26
bazel_0_29
bazel_1
bazel_3
belle-sip

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/kdy4kbxfi8m69cb4xh98h91fbj55g5am-melonDS-0.9.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A melonDS https://github.com/r-ryantm/nixpkgs/archive/9b45a8e56553dea33f406aef53e6a9efbf7ec482.tar.gz

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/kdy4kbxfi8m69cb4xh98h91fbj55g5am-melonDS-0.9.1
ls -la /nix/store/kdy4kbxfi8m69cb4xh98h91fbj55g5am-melonDS-0.9.1/bin

Pre-merge build results
NixPkgs review skipped

Maintainer pings
cc @artemist @benley @SCOTT-HAMILTON for testing.

This builds and runs a NDS ROM on my x86_64 workstation but I haven't tested it on AArch64

This builds and runs a NDS ROM on my x86_64 workstation but I haven't tested it on AArch64


Suppport a Lingy nREPL server

We need the monolithic Lingy REPL to use a Clojure-like client server model to allow other UIs / IDEs to support Lingy.
@PEZ, can you give insight about how Clojure does this?

There is some risk I will be saying some untrue things, but I can jot down how things work to the best of my knowledge.
In the blog post shadow-cljs + Clojure with Calva: The basics, and when interviewed by the Youtube channel On the Code again, I go into some discussion around what the REPL is and how it relates to nREPL. There is also some info about this here: https://calva.io/nrepl_and_cider-nrepl/
Nevertheless, I'll attempt a quick summary here... 😄
What's the REPL?
Let's do the flashy-flashy thingy on a thing first: The REPL is not a prompt. The REPL is a process running the application under development. The REPL prompt is a UI to the REPL.
When we develop Clojure apps the Clojure Way, we generally do not use the write -> compile -> start -> stop -> write -> compile -> start ... cycle that is common for many programming environments. Instead we:

Start the REPL (remember, we could use the REPL prompt for this, but often we use other means)
We use the REPL to compile our application (or part of it) and start it. Starting the application can mean different things, but at the very least the REPL keeps running.
We send bits of existing and new pieces of our application to the REPL, which compiles and evaluates them. Such an evaluation can result in that we get information about the running application back, or that something in the application gets redefined, or created.

We then mostly stay in step 3 there. See https://www.youtube.com/watch?v=d0K1oaFGvuQ for a demo of solving FizzBuzz this way. Or https://www.youtube.com/watch?v=5ZyAhGv55V0 for a (much longer) demo of working with a mobile app this way. Some of these REPL sessions can last for weeks without us ever restarting the application.
What's nREPL?
For the purpose of this ticket nREPL is a protocol for talking to the REPL, enabling both prompts and other UIs to the REPL. Today Lingy has one UI to its REPL, the prompt, which “talks” directly to the Lingy REPL.
Clojure also has a prompt like this, and in addition it has a Socket REPL, which tooling can use to create REPL UIs. However, the Socket REPL is unstructured, and a bit tricky to use, so most tooling use nREPL, which is a community library (and has been a bit controversial for reasons I don't quite understand, but I think that these days most people just accept that it is ubiquitous).
nREPL is a network client/server protocol, most often (maybe always? running over TCP). The server can support one or more transport protocols. Most clients use Bencode transport. The nREPL server also supports EDN and TTY (telnet, etc) transports.
The nREPL protocol has the client sending messages to the server and getting responses back. A message is a struct/map with at least a handler name (op). nREPL defines a few standard ops, that most servers and clients are supposed to support. It is extensible via middleware to support other ops. The most popular middleware is cider-nrepl.
Here's some sample output from Calva with nREPL message tracing on:
-> sent
{
  op: 'info',
  ns: 'test-lein.core',
  symbol: 'println',
  id: '7',
  session: '1a080b66-b1b6-4b8c-8206-c4af2cc02747'
}

<- received
{
  added: '1.0',
  'arglists-str': '[& more]',
  column: 1,
  doc: 'Same as print followed by (newline)',
  file: 'jar:file:/Users/brandon/.m2/repository/org/clojure/clojure/1.10.1/clojure-1.10.1.jar!/clojure/core.clj',
  id: '7',
  line: 3733,
  name: 'println',
  ns: 'clojure.core',
  resource: 'clojure/core.clj',
  'see-also': [
    'clojure.core/prn',
    'clojure.core/print',
    'clojure.core/println-str',
    'clojure.pprint/pprint'
  ],
  session: '1a080b66-b1b6-4b8c-8206-c4af2cc02747',
  static: 'true',
  status: [ 'done' ]
}

The info message is sent when the user hovers a Clojure symbol in VS Code. In the example we can see that apart from the op, each message also refers a session and has an id. When the nREPL client connects it gets a new session. Calva then clones this session first thing, using the clone op.
An nREPL session corresponds to a set of bindings for some dynamic variables, like ns and some more. So even if two nREPL sessions are manipulating the same application, they bind and re-bind the dynamic variables separately.
The message id is used by the client to connect responses to messages it has sent.
Apart from the standard message fiels, op, id, and session (and possibly ns, I don't recall right now), each op supports different fields for the both requests (which I don't think are called requests, but I could be wrong) and the responses.
It's getting late here. I'll reread this tomorrow and see if I agree still. And if I think something is missing. Please don't hesitate to ask questions, @ingydotnet !

@PEZ Thanks for the detailed explanation.
As we discussed in Slack, I'll create an nrepl branch that adds lingy --nrepl CLI support.
When run this will start a TCP socket server and print nrepl://<IP_ADDRESS>:45373 to stdout.
We can iterate on what needs to happen from there.


ci: refactor using a single steps workflow

The logic for the GHA steps was repeated across multiple workflows and platforms. Consolidate it all into a common "ci-steps" workflow, and have all the other workflows reference that when possible. This should hopefully make maintenance of the CI scripts easier in the future, with less redundancy of the step logic.
The only major place where we deviate is that we have a separeately specified set of steps for the "old" VFX Platform 2021 and 2022 based tests (because we need to specify different container arguments to it).

Any opinions on this?
Again what's happening here is that we're merging all the (mostly redundant) step lists of the various workflows into a single step list, then referencing it from the other places. Flags are used to skip the steps for workflows where they are not needed (for example, the "build docs" workflow skips actually building or testing the code). Mostly, it's just eliminating redundancy and the possibility of the different workflows diverging and doing different things even for steps that we intend to be the same across all of them workflows.

This has been sitting around for a while without any comment. I will merge it by the end of the week if there are no objections.


Added compress option to dnsmasq log files

Description

This PR adds compression to dnsmasq log files

Fixes: #3756
Types of changes

[ ] Breaking change (fix or feature that would cause existing functionality to change)
[ ] New feature (non-breaking change which adds functionality)
[ ] Bug fix (non-breaking change which fixes an issue)
[x] Enhancement (improves an existing feature and functionality)
[ ] Cleanup (Code refactoring and cleanup, that may add test cases)

Screenshots (if appropriate):
How Has This Been Tested?

Log into virtual router
Check /var/log/ directory, observe dnsmasq.log
Change system time 1 day ahead  and reboot VR
Check /var/log/ directory, observe dnsmasq.log, dnsmasq.log1
Change system time 1 day ahead  and reboot VR
There should be 3 log files now and 1 should be compressed
Check /var/log/ directory, observe dnsmasq.log, dnsmasq.log1, dnsmasq.log.2.gz

@blueorangutan package

@Spaceman1984 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

@blueorangutan package

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2325

@blueorangutan test

@Spaceman1984 a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

Trillian test result (tid-3119)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 31996 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr4439-t3119-kvm-centos7.zip
Smoke tests completed. 86 look OK, 0 have error(s)
Only failed tests results shown below:

Test
Result
Time (s)
Test File

Simple enough change, let's merge for master @DaanHoogland ?

@PaulAngus merging this as it is small and not intrusive. cc @GabrielBrascher @weizhouapache.


Home Page

👷 Deploy Preview for crpp-ai-oncology processing.

Name
Link

🔨 Latest commit
74c1092138d1eea8c4fc02631e5f116e5b6f5ae0

🔍 Latest deploy log
https://app.netlify.com/sites/crpp-ai-oncology/deploys/636b943631b13f0008090a56


Sort techniques drop-down

As a user, when I go and select a technique from the drop-down, I would expect to found them ordered alphabetically.
It would make it easier to find the correct one.

This is already implemented but not yet deployed :+1: - https://github.com/panosc-eu/searchui/blob/main/src/Search/AsyncAutocomplete.js#L12
I will try to work on continuous deployment to GitHub Pages next week.

Deployed branch new-numeric-param that also includes the sorted techniques.


Feature Request: Tracker Checker and Ratio

So I'm working on building a website and I want to make it basically a private tracker that runs on webtorrent and yes I do know that WT always requires DHT but I mean more in the sense of ratios and being able to organize all my torrents. So how difficult would it be or how long would it take before you can add 1: ratios to the desktop application. and 2: add a menu that shows the response and peer count of each tracker, ie
wss://webtorrent.io:80 - announce ok - 14 Seeders 7 Leechers
Or does the WT tracker machine not operate with normal tracker announce requests and responses?
Also to avoid making another issue, how possible or efficient would it be to make a seedbox style version of WT Desktop, so instead of being able to stream it operates and appears similar to a regular BitTorrent client and can manage trackers and record ratio and so forth. Cause the CLi version can be cumbersome when it comes to seeding and having to make a new window for every torrent added.

It's really a lot easier to track progress on each individual issue you mentioned if you make an issue for each topic.

how long would it take before you can add ratios to the desktop application

Do you mean per-torrent ratios? Or a global ratio? Honestly, this doesn't seem useful for 90% of users, so it's not a top priority. Especially if it complicates the UI. But there may be a subtle way to show it.

add a menu that shows the response and peer count of each tracker, ie

This seems even more specific. I don't think we'll add this.

Or does the WT tracker machine not operate with normal tracker announce requests and responses?

There are normal tracker announces and responses. The underlying webtorrent library does expose this information: https://github.com/feross/webtorrent

it operates and appears similar to a regular BitTorrent client and can manage trackers and record ratio and so forth

I saw another project trying to build a more traditional torrent client UI using webtorrent. https://github.com/FaCuZ/torrentmedia Not sure what the status or progress on that is. Maybe you can help that effort out?
Honestly, though, the underlying webtorrent library does expose the information and functionality you're looking for, so I would consider writing some code to get exactly what you want.

Ok thank you for pointing me out to that project. The reason I was hoping for these features, was that I wanted to be able to test the connection of my tracker and DNS services once I get a domain on the IP. I was hoping that with that I'd be able to setup my tracker and discern my uptime. Cause using the NodeJSCli to run the tracker while that is all fine and easy, the problem is that I cannot seem to find a method that confirms the WSS:// tracker is functioning properly since the Desktop version and the browser versions don't display announces. Come to think of it since you said the announces are similar would it be able write a quick segment of JS up so that it'll display the announces in the browser for all the trackers?
That's the only concern I have with the tracker announcing. Because as of now half the trackers could be dead and no one knows cause how could they. As well as the tracker i'm setting up could be improperly configured while I go sharing it anyway. Maybe if that is possible adding it to the /docs page of the website.
But the ratio was less important and was hoping just to have it added in the little corner to the right so it's out of the way but visible. It seems like it would take longer to build a global ratio.

You can check that your wss tracker is working by enabling wss only and then visiting it's /stats page. For example, this is what it looks like in practice on a community-run tracker: https://tracker.btorrent.xyz/stats


Possible missing config line?

As I was editing PJC.conf, I noticed the "database=" line was missing from "settings.sql".  That line is present in "pjp.conf", but not in pjc
I added it to my file just in case.

So the sql settings in pjc are just connection settings that my other plugins to use. pjc doesn't have database which is why the key is not there. pjp should only contain database key under the sql node. Basically each plugin uses the connection settings from pjc, but each plugin uses it's own database, so it's specified in the plugin config. Hope that makes sense
PJC:
settings.sql.enable=false
settings.sql.password=password
settings.sql.prefix=NONE
settings.sql.url="localhost:3306"
settings.sql.username=root
PJP:
settings.sql.database=pjp

It does.  I'll remove the line I added.


Crawling from some URLs is not possible

Crawling some urls with the following configuration (see below) works the crawler just fine. But with a few common urls it gives unexpectedly the error message (The real url name is intentionally changed):
ERROR - Could not extract links from:  some-url.com
      java.io.UnsupportedEncodingException: IBM420_ltr
	at java.base/sun.nio.cs.StreamDecoder.forInputStreamReader(StreamDecoder.java:71)
	at java.base/java.io.InputStreamReader.<init>(InputStreamReader.java:100)
	at com.norconex.collector.http.url.impl.GenericLinkExtractor.extractLinks(GenericLinkExtractor.java:337)
	at com.norconex.collector.http.pipeline.importer.LinkExtractorStage.executeStage(LinkExtractorStage.java:73)
	at com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:31)
	at com.norconex.collector.http.pipeline.importer.AbstractImporterStage.execute(AbstractImporterStage.java:24)
	at com.norconex.commons.lang.pipeline.Pipeline.execute(Pipeline.java:91)
	at com.norconex.collector.http.crawler.HttpCrawler.executeImporterPipeline(HttpCrawler.java:360)
	at com.norconex.collector.core.crawler.AbstractCrawler.processNextQueuedCrawlData(AbstractCrawler.java:538)
	at com.norconex.collector.core.crawler.AbstractCrawler.processNextReference(AbstractCrawler.java:419)
	at com.norconex.collector.core.crawler.AbstractCrawler$ProcessReferencesRunnable.run(AbstractCrawler.java:812)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)

In this case crawler ends very fast
 INFO -            URLS_EXTRACTED: http://some-url.com/
 INFO -       REJECTED_UNMODIFIED: http://some-url.com/
 INFO - some-url.com: Deleting orphan references (if any)...
 INFO - some-url.com: Deleted 0 orphan references...
 INFO - some-url.com: Crawler finishing: committing documents.
 INFO - Elasticsearch RestClient closed.
 INFO - some-url.com: 1 reference(s) processed.
 INFO -          CRAWLER_FINISHED
 INFO - some-url.com: Crawler completed.
 INFO - some-url.com: Crawler executed in 0 second.
 INFO - some-url.com: Closing sitemap store...
 INFO - Running some-url.com: END (Wed Feb 28 09:27:18 CET 2018)

That is the crawler's configuration I use:
<httpcollector id="Company-Crawler">
  #set($workdir = "/opt/norconex/http-collector/test-output")
  #set($core      = "com.norconex.collector.core")
  #set($http      = "com.norconex.collector.http")
  #set($committer = "com.norconex.committer")

  #set($httpClientFactory = "${http}.client.impl.GenericHttpClientFactory")
  #set($filterExtension   = "${core}.filter.impl.ExtensionReferenceFilter")
  #set($filterRegexRef    = "${core}.filter.impl.RegexReferenceFilter")
  #set($filterRegexMeta   = "${core}.filter.impl.RegexMetadataFilter")
  #set($robotsTxt         = "${http}.robot.impl.StandardRobotsTxtProvider")
  #set($robotsMeta        = "${http}.robot.impl.StandardRobotsMetaProvider")
  #set($urlNormalizer     = "${http}.url.impl.GenericURLNormalizer")
  #set($esCommitter       = "${committer}.elasticsearch.ElasticsearchCommitter")
  #set($nodesUrl 		  = "http://<IP_ADDRESS>:9200")

  <progressDir>${workdir}/progress</progressDir>
  <logsDir>${workdir}/logs</logsDir>

  <crawlerDefaults>
	<userAgent>Norconex Collector-Http</userAgent>
	<urlNormalizer class="$urlNormalizer" />
	<numThreads>3</numThreads>
	<maxDepth>5</maxDepth>
	<workDir>$workdir</workDir>
	<keepDownloads>false</keepDownloads>
	<orphansStrategy>DELETE</orphansStrategy>
	<robotsTxt ignore="true" />
	<robotsMeta ignore="true" />
	<sitemapResolverFactory ignore="true" />

	<referenceFilters>
		<filter class="$filterExtension" onMatch="exclude">jpg,gif,png,jpeg,bmp,pdf,ico,css,js,sit,eps,wmf,ppt,mpg,mp4,xls,rpm,mov,exe</filter>
    </referenceFilters>
  </crawlerDefaults>

 <crawlers>
	<crawler id="some-url.com">
		<startURLs stayOnDomain="true" stayOnPort="false" stayOnProtocol="false">
			<url>http://some-url.com</url>
		</startURLs>
		<committer class="$esCommitter">
			<nodes>$nodesUrl</nodes>
			<discoverNodes>true</discoverNodes>
			<indexName>suchindex</indexName>
			<typeName>webdoc</typeName>
			<targetContentField>body</targetContentField>
    	  	<queueSize>100</queueSize>
  			<commitBatchSize>500</commitBatchSize>
			<ignoreResponseErrors>true</ignoreResponseErrors>
		</committer>
    </crawler>
  </crawlers>
</httpcollector>

I use the latest version of crawler and java version I use is:
Java(TM) SE Runtime Environment (build 9.0.4+11)
Please give me an advice on how to set the crawler to fix this error.
Thank you.

Without a URL I can't reproduce, but from the error, it seems that the encoding detected is not supported by your JRE (IBM420_ltr).
Short of changing the encoding of the page causing the issue, you can try to force the link extractor to read links as a different encoding.  For example, you can try adding the following under your crawler configuration to use UTF-8:
<linkExtractors>
 <extractor class="com.norconex.collector.http.url.impl.GenericLinkExtractor" charset="UTF-8"/>
</linkExtractors>

That was exactly the problem. Your advice has fixed my problem. Thank a lot Pascal :+1: