Adjust tile rank for transit stations per railway type

In London the kind_tile_rank sometimes biases towards lower quality service features like light rail & trams to the detriment of larger stations serving "heavy" rail. The same thing is sometimes true in New York (comparing Manhattan to New Jersey when suddenly many light rail appear when only heavy rail appear in Manhattan). There are probably two factors at work here: tile versus meta tile, and railway type. This issue is focused only on railway type.
Since OSM doesn't have trip frequency level data, the route relations act as a good proxy, but the fact that something has more physical infrastructure investment should also act as a signal that there are probably more trips passing thru a station.
On railway type, from @zerebubuth:
If we want to count subway routes separately from overground routes,
then we can do that. Many (although not all) route relations are tagged
to say whether they're surface, subway, light rail or tram lines. It just
seems like this makes comparing stations harder (i.e: which is more
important, the one with 4 surface lines and 2 subway lines, or the one
with 2 surface lines, 1 light rail line and 3 tram lines?).

Proposed solution is to determine the "preferred" / "best" / "dominant" transit type for a station, and to also sort based on that as a tie breaker, maybe weight it, add zooms to the mix?

heavy rail station A with 2 lines, light rail station with 2 lines.

not

light rail station A with 2 lines, heavy rail station with 2 lines.

London
12/51.5093/-0.0760

One possible way to do it, which feels like it might be quite intuitive, is to take:

The number of mainline rail routes, up to a maximum of 9 and multiply by 100.
The number of subway plus light rail routes, up to a total maximum of 9 and multiply by 10.
The number of tram / other routes, up to a maximum of 9.

Sum these, and we'll get a score for each station between 0 and 999 (it would be interesting to see if any station in the world gets 999). That can then be the value used for sorting to produce the final rank.
Alternatively, if we think that something with 9 light rail routes should be more important that a single mainline route, then we could add them together after scaling, say 6 * mainline + 2 * light + others or something to that effect.
Finally, a few observations about London:

What's that "tear" near West Silvertown near the river to the east?
Almost no stations are showing in the centre of London - are they being suppressed by the road labels? Even if the correct stations aren't showing, I'd expect some stations to show.
I forgot that in production we use the "flat nodes" functionality of osm2pgsql, so raw nodes aren't stored in the database (on my local machine, I don't use that function), so I'll need to patch the function to try joining on the planet_osm_point table instead. Tracking that in #507.

I dig the proposed solution, let's take it for a spin.
Digging into your comments about London:

The "tear" is, I think, an areal way with some Tangram rendering artifacts. Brett is already on it (we saw similar in Hong Kong a couple weeks ago).
Road labels don't interact with station labels right now (station labels are "on top of" road labels). It looks like it's being compounded by a filter that worked in the USA but not in London: if the station doesn't have an "area" then it's considered less important. Maybe that should be a contributing factor server side influencing the kind_tile_rank, but it doesn't seem useful by itself client side?

Some images (below), with kind_tile_rank labeled and stations below the threshold shown tiny icons (and the area filter removed, thus restoring Waterloo and a couple more stations).
Right now subway neighbouring stations that share a couple lines for a small run pop out and get a better rank. True both for subway and light rail – and how there are several light rail routes so they're blowing out nearby subway stations with just 1 or 2 routes.

I think the fix in #508 will have a big effect here.
Re: areas, I think that's a good tie-break rule, and it'll be more effective once #348 is done. For the moment, it might do more harm than good, as there's a lot of major stations mapped as nodes - Penn Station in NYC, for example.

Looks like things are better (thanks to #508), but big rail stations are still missing / too low of a rank. Let's pursue the options @zerebubuth describes above for this milestone.
Testing scene file: tangram-skin-and-bones.zip
The same screenshot areas as above:

We had been talking in another issue about setting the kind to be more specific (subway, light_rail, etc not just station). Seems like the lookup process would be the same, should we store a new "service" property on the feature as part of the tile_kind_rank calculation?

A couple of points here:
London isn't a great place to be testing public transit stuff in OSM. When I was exploring the data, it wasn't one of the cities with the most well-mapped public transit relations. Paris or Berlin would be better, and perhaps we can set aside some time to fix the data in London / NYC?
Public transit relations often span the overground and underground (subway) stations, as well as having light rail and/or international services. This means that a single kind or service tag doesn't fully represent the services at a station. A couple of possible solutions to that:

a service hierarchy (which would limit styling options in the case that the user prioritises subway / light rail over international services)
or a services=[...] list (which isn't supported by all formats)

A third option would be to add boolean flags, e.g: is_light_rail, is_subway, is_international. The downside of that is that it's quite verbose, but it is supported by all formats and railway stations are fairly rare features.

Here are the top 20 stations (or halts, tram stops, aerialway stations) in London according to the scoring system set out in this comment:
933	n3662847634	station	"London St. Pancras"
933	n3637436408	station	"London Liverpool Street"
933	n119274464	station	"King's Cross St. Pancras"
631	n441947584	station	"Victoria"
631	n3768010552	station	"London Victoria"
610	w140162700	station	"Stratford"
610	w140162700	station	"Stratford"
600	w170130337	station	"West Ham"
510	w72570259	station	"Queen's Park"
510	n3663368461	station	"Euston"
510	n3663368460	station	"London Euston"
500	w25025418	station	"Canning Town"
431	w302026559	station	"London Paddington"
431	r204439 	station	"London Paddington"
431	n1264482797	station	"Paddington (District, Circle, and Bakerloo Lines)"
420	w289193423	station	"Whitechapel"
420	n2426825788	station	"Whitechapel"
410	w198353814	station	"Highbury & Islington"
401	n3757075887	station	"Wandsworth Common"
400	w41599363	station	"Canada Water"

For comparison, here are the ones that I wrote down before starting that I'd expect to see at the top (but in no particular order). The bold ones are not in the above top 20, and show where they are in the list:

Paddington
Waterloo (114th / 612)
Victoria
London Bridge (233rd / 612)
Liverpool Street
King's Cross / St. Pancras
Euston
Marylebone (59th / 612)
Farringdon (560th / 612)

The remaining issues seem to be mostly data:

Only one of Waterloo's mainline rail routes is detected, because only one is mapped (the yellow highlight on ÖPNVKarte).
None of London Bridge's mainline routes are detected, for a similar reason.
Marylebone detects a couple of mainline routes and a subway route, and comes in at 59th - probably just needs more routes mapping, as I'm pretty sure it serves more than 2 routes.
Farringdon gets a score of zero because the transport relations are broken and there's no way to "get to" the routes from the station node.

I've played around with the scoring function a little; my intuition is that an interchange station is more important than something which either only serves rail routes or subway routes. But it seems like it's hard to quantify.

I should add: there are duplicates in that list, but we'll try to get rid of those "in post" using the site or stop_area relations which connect them.

Yes! I was thinking the same thing in https://github.com/mapzen/vector-datasource/issues/587 regarding interchange stations (and even terminal stations). Double their score?
If one of these is a "transfer" station where two lines come intersect (but not in a sequence
running parallel), I could see zoom 12. And if it's the terminal station on the line then zoom 11.

Here's the top 20 with the first two digits doubled if they're both non-zero:
963     n3662847634     station "London St. Pancras"
963     n3637436408     station "London Liverpool Street"
963     n119274464      station "King's Cross St. Pancras"
961     n441947584      station "Victoria"
961     n3768010552     station "London Victoria"
920     w72570259       station "Queen's Park"
920     w140162700      station "Stratford"
920     w140162700      station "Stratford"
920     n3663368461     station "Euston"
920     n3663368460     station "London Euston"
861     w302026559      station "London Paddington"
861     r204439  station "London Paddington"
861     n1264482797     station "Paddington (District, Circle, and Bakerloo Lines)"
840     w289193423      station "Whitechapel"
840     n2426825788     station "Whitechapel"
820     w198353814      station "Highbury & Islington"
623     n307542330      tram_stop       "Wimbledon"
623     n18089211       station "Wimbledon"
621     r3791635        station "Willesden Junction"
620     n18036791       station "Poplar"

It's very similar to the last one, except we've lost West Ham, Canning Town, Wandsworth Common and Canada Water and gained Wimbledon (x2), Willesden Junction and Poplar. Which, on the whole, seems like an improvement.
I'll start porting this to SQL so we can try it out on dev.

Related: add osm_site_relation: https://github.com/mapzen/vector-datasource/issues/590

Related: #587.

Progress dump.
Note: For the screenshots below, I had to turn off all filtering of stations by area, and drop the min_zoom for all station queries to 10 (see 1a2f07a7233b67969f12f0ce0d00ce0fdc61e709). This might not be such a bad thing, as stations are relatively rare and we can drop them based on kind_tile_rank before the client ever sees them at lower zooms. Alternatively, we can base the min_zoom calculation on the transit score instead.
London, z11

London, z12

Top 20 data:
London
 typ  | osm_id   | score | root_relation_id |           name
------+------------+-------+------------------+--------------------------
 node |  441947584 |   961 |          1572442 | Victoria
 node |<PHONE_NUMBER> |   961 |          1572442 | London Victoria
 way  |   72570259 |   920 |          3803730 | Queen's Park
 node |<PHONE_NUMBER> |   920 |          1569858 | London Euston
 node |  963506307 |   920 |           199275 | Wembley Central
 way  |  140162700 |   920 |           297366 | Stratford
 node |   10287073 |   920 |           199275 | Wembley Central
 node |<PHONE_NUMBER> |   920 |          1569858 | Euston
 way  |  289193423 |   840 |          3155934 | Whitechapel
 node |<PHONE_NUMBER> |   840 |          3155934 | Whitechapel
 way  |  198353814 |   820 |           206255 | Highbury & Islington
 node |<PHONE_NUMBER> |   663 |          1569873 | London St. Pancras
 node |  119274464 |   663 |          1569873 | King's Cross St. Pancras
 way  |    4959629 |   663 |          1571791 | Liverpool Street
 node |<PHONE_NUMBER> |   663 |          1571791 | London Liverpool Street
 way  |   -3791635 |   621 |          3791636 | Willesden Junction
 way  |   -3791635 |   621 |          3791636 | Willesden Junction
 node |   18036779 |   620 |                  | Westferry
 node |   18036791 |   620 |                  | Poplar
 node |   18020039 |   620 |                  | Limehouse

A couple of odd things here:

The high rank of Queen's Park is because it has a part-time stop on a national rail route. As far as I can tell, this "part-time" is basically never, so this might be considered a data issue. Wembley is similar: the station node is attached to the railway, along which these national rail routes run, but they don't actually stop at Wembley. One fix for this might be stricter about the roles via which routes are discovered, but this would have unwanted side-effects on other stations.
The high rank of the DLR stations (Westferry, Poplar, Limehouse) is because 3 of the DLR routes were tagged as being rail routes. I've since fixed the data to make them light_rail routes instead.

NYC
 typ  |   osm_id   | score | root_relation_id |                       name
------+------------+-------+------------------+---------------------------------------------------
 node |  895371274 |   902 |                  | New York Penn Station
 way  |  265947358 |   302 |          4637816 | Grand Central Terminal
 node |<PHONE_NUMBER> |   260 |                  | 14th Street (1,2,3)
 node |  597928313 |    70 |                  | 59th Street - Columbus Circle (A,B,C,D,1)
 node |  544356374 |    70 |                  | 34th Street-Herald Square (B,D,F,M,N,Q,R)
 node |<PHONE_NUMBER> |    60 |          2917356 | 59th Street (4,5,6)
 node |  597928309 |    60 |                  | West 4th Street-Washington Square (A,B,C,D,E,F,M)
 node |  597928318 |    60 |          2917356 | 59th Street-Lexington Avenue (N,Q,R)
 node |  591998010 |    50 |          2917492 | 53rd Street-Lexington Avenue (E,M)
 node |  597928319 |    50 |          2917492 | 51st Street (6)
 node |  597928317 |    50 |                  | 23rd Street (F,M,PATH)
 node |  591997657 |    50 |                  | 42nd Street-Grand Central (S,4,5,6,7)
 node |  597928308 |    50 |          3420745 | 14th Street (F,M,PATH)
 node |  597928312 |    40 |                  | Delancey Street-Essex Street (F,J,M,Z)
 node |  597928316 |    40 |                  | 42nd Street - Bryant Park (B,D,F,M)
 node |  597928314 |    40 |                  | 47th-50th Streets - Rockefeller Center (B,D,F,M)
 node |<PHONE_NUMBER> |    40 |                  | Fulton Street-Broadway Nassau (A,C,J,Z)
 node |<PHONE_NUMBER> |    40 |          3420790 | 14th Street-8th Avenue (A,C,E,L)
 node |  591995518 |    40 |                  | Broadway-Lafayette Street (B,D,F,M)
 node |<PHONE_NUMBER> |    40 |          3420670 | 14th Street-Union Square (4,5,6)

NYC doesn't have many station relations, so it looks like it's mostly sorting the mainline stations first. The only interchange station is 14th Street, because of this suspect-looking relation. @rmarianski - are there any non-subway routes running through 14th Street?
San Francisco
 typ  |   osm_id   | score | root_relation_id |                  name
------+------------+-------+------------------+-----------------------------------------
 way  |   28295229 |   200 |                  | San Francisco 4th & King Street Station
 node |<PHONE_NUMBER> |   100 |                  | 22nd Street
 node |<PHONE_NUMBER> |   100 |                  | Bayshore
 node |<PHONE_NUMBER> |    60 |                  | Embarcadero
 node |  297863037 |    60 |                  | Van Ness Muni
 node |  313885839 |    40 |                  | West Portal Muni
 node |  297863017 |    40 |          1069465 | Castro Muni
 node |  301506011 |    40 |          1069835 | Church St Muni
 node |<PHONE_NUMBER> |    40 |                  | West Portal Muni
 way  |  159782838 |     0 |                  | Balboa Park
 way  |  101546361 |     0 |                  | Daly City
 way  |  136822046 |     0 |                  | Forest Hill
 node |<PHONE_NUMBER> |     0 |                  | 24th Street Mission
 way  |  132246043 |     0 |                  | Glen Park
 node |<PHONE_NUMBER> |     0 |                  | Civic Center / UN Plaza
 node |  301506348 |     0 |                  | Forest Hill Muni
 node |<PHONE_NUMBER> |     0 |                  | Powell Street
 node |<PHONE_NUMBER> |     0 |                  | Embarcadero
 node |<PHONE_NUMBER> |     0 |                  | Montgomery Street
 node |<PHONE_NUMBER> |     0 |                  | 16th Street Mission

There's a lot of opportunity for improvement in San Francisco!

Starting at zoom 10 makes sense to me, and ties in well with other changes in this release related to subway line zoom ranges.
Why aren't there more subway stations visible in central London at zoom 11 and zoom 12? For instance, many of the stations in this screenshot off prod seem to be missing (like Barbican)?

I think the 14th St station in NYC correctly includes the PATH commuter rail / other subway like line's 14th St station?

@zerebubuth should we include root_relation_id=* instead of osm_relation:yes?

For instance, many of the stations in this screenshot off prod seem to be missing (like Barbican)?

I can see Barbican - it's about 2/3rds of the way from the left of the image and halfway down it, just above the London Wall neighbourhood label and to the left of the Barbican neighbourhood label.
Why are we showing Aldwych? (It's the one on the dark blue Piccadilly line spur from Holborn) Both the line and the station have been closed for more than 20 years.

Why are we showing Aldwych? (It's the one on the dark blue Piccadilly line spur from Holborn) Both the line and the station have been closed for more than 20 years.

Looks like theres a couple of things going on here...
Aldwych station is being shown because it has disused:railway=station, which doesn't match any known POI type, so it's being put in the buildings layer. Because it also has building=station, then the label placement for it has kind:station. This means we probably should either:

Not be displaying POIs on the basis of what's in the buildings layer - that's what the POIs layer is for, and/or
Resetting tags on disused:* elements.

I would prefer the former, as I'm pretty sure that the latter will result in some strange and difficult-to-track-down bug when there's a disused:power=plant which becomes a tourism=gallery for example, and we end up suppressing it.
The reason that the spur is appearing on the map is because the disused railway is a member of the route relation. There's a good case why it shouldn't be, but it is tagged railway=disused, so we would be able to filter it out with some changes to osm2pgsql.

Yeah, I see Barbican on my colorful screenshot, but not in your London screenshots from zoom 11 and 12?
Good news, looks like Aldwych (Closed) station will be dropped once https://github.com/mapzen/vector-datasource/issues/483 lands:

Oh, Barbican is there - I just stopped the style from displaying anything with kind_tile_rank>16 because it was getting too cluttered. At z12, Barbican is in tile 12/2046/1361, which covers a huge chunk of central  London, so a rank of 20 doesn't seem unreasonable:

Good news, looks like Aldwych (Closed) station will be dropped once #483 lands

Hmm... how about Great Falls Depot or the old King's Cross Pentonville Road Entrance?

Oh, Barbican is there - I just stopped the style from displaying anything with kind_tile_rank>16 because it was getting too cluttered.

Cool, looking forward to seeing this on dev.

Not be displaying POIs on the basis of what's in the buildings layer - that's what the POIs layer is for, and/or

Aha, good catch! Looks like that's a long standing bug in the scene files where we were displaying icons for building label placement as if they were POIs (and we shouldn't):

https://github.com/nvkelso/tangram-skin-and-bones/issues/24
https://github.com/tangrams/eraser-map/issues/168

Here's a preview:

Hmm... how about Great Falls Depot or the old King's Cross Pentonville Road Entrance?

Agree, here's the above solution applied to these two buildings:

Change is looking fantastic in NYC, London, Paris! As you note, SF is so-so but that's because it needs more linking-up on the data side. There aren't any regressions over v0.8 prod so we're good.
One question:
I expected two Waterloo stations, one for rail, one for subway in zoom 16 tile, but I don't see the subway station<PHONE_NUMBER>). Are we too aggressively de-duplicating features? Perhaps they split apart at zoom 17 when originally implemented in the earlier release, but now that we're "meta-tiling" at zoom 16 we need to adjust the queries.yaml end_zoom? Starting at zoom 14 I expect to see individual features. Is it as simple as queries.yaml, or does other logic need to change?

https://www.openstreetmap.org/edit?node=3638795617#map=19/51.50284/-0.11280
https://www.openstreetmap.org/edit?node=3638795618#map=19/51.50299/-0.11396

I think zoom 14 is far too early to start showing individual features. At zoom 14, the two Waterloo station points are 14px apart, which is just enough for two confusingly tiny 12px icons to be drawn (but neither labelled). Even if one had a label, it would be confusing as to which point was being labelled.
Zoom 15 is not much better, and I think it's probably best to split the features at z16. I'm not quite sure whether you're saying that's never happening since we stopped generating >z16 tiles or not?

Sure, let's give zoom 15 try (to stop dedup'ing). There's certainly room to place the multiple icons by then, and it looks weird / broken to not show them when everyone else does (labeling is labeling, shrug). Looks like the same problem is happening at Euston?

Everything else looks good, and this isn't causing a regression. Let's pick up the zoom 15+ stuff in the next release with https://github.com/mapzen/vector-datasource/issues/637.


⚠ AGEG website (prod) has degraded performance

In 805e65b, AGEG website (prod) (https://www.ageg.ca/) experienced degraded performance:

HTTP code: 200
Response time: 6749 ms

Resolved: AGEG website (prod) performance has improved in fdd5bf5.


chore(v2): remove alpha 58 doc

Motivation
It's been more than 3 months since 58 ver was released, so we can remove its docs.
Have you read the Contributing Guidelines on pull requests?
Yes
Test Plan
Preview.
Related PRs
(If this PR adds or changes functionality, please take some time to update the docs at https://github.com/facebook/docusaurus, and link to your PR here.)

Yep, thank you, Yangshun, I feel so much better now! :)
P.S. check your Discord please.


New command: teams message remove

Usage
m365 teams message remove [options]
Description
Removes a message from a channel in a Microsoft Teams team
Options

Option
Description

--teamId [teamId]
ID of the Microsoft Teams team. Specify either teamId or teamName but not both.

--teamName [teamName]
Name of the Microsoft Teams team. Specify either teamId or teamName but not both.

--channelId [channelId]
Channel ID of the Microsoft Teams team. Specify either channelId or channelName but not both.

--channelName [channelName]
Channel name of the Microsoft Teams team. Specify either channelId or channelName but not both.

-i, --id <id>
The ID of the Teams message.

-f, --force
Don't prompt for confirmation.

Examples
Remove a message by using IDs
m365 teams message remove --teamId 5f5d7b71-1161-44d8-bcc1-3da710eb4171 --channelId<EMAIL_ADDRESS>--id<PHONE_NUMBER>203

Remove a message by using display names
m365 teams message remove --teamName Marketing --channelName Branding --id<PHONE_NUMBER>203

Default properties
No response
Additional Info
Remarks:

This command doesn't support application permissions, let's check for this

API: https://learn.microsoft.com/en-us/graph/api/chatmessage-softdelete?view=graph-rest-1.0&tabs=http

LGTM 👍 let's do it 🚀

Can I work on this?

@milanholemans I have noticed that we use an interface ExtendedGroup to retrieve the Team and confirm that we are trying to execute an action on a Microsoft 365 Team. However, we use the same piece of code for this across 12 commands.
Would it be handy if I put this into a util (or extend the entraGroup util)?

I think it would make sense to centralize ExtendedGroup instead of copy/pasting it over and over again. Good catch!
About the retrieval of Teams groups and channels by name. I'm all into centralizing this as well. This enables us to throw a uniform error in multiple commands and introduce interactivity when multiple teams with the same name exist. Let's include this in a new teams util. Great suggestions 👍

Hi @milanholemans
I have noticed that currently, we haven't got enough permissions to execute the request to the Graph API, as we require the ChannelMessage.ReadWrite permission to be able to delete a message in a channel.
Is it possible that it gets added to the PnP Application Registration? In the meantime, I will continue developing using my own application.

Yes, be sure to mention this in your PR so we don't forget it.

@milanholemans I think for the refactoring of all the other commands, it might be handy that we create a different issue for this. Otherwise, this PR will become too big I feel.
In that issue, we could possibly also refactor all the commands so that they support both teamId and teamName, and channelId and teamName, as currently this is quite inconsistent.
Does that seem fine for you?

Makes sense. Let's start with this command, and then continue with the rest I suppose 😄


Ability to export to coreml models for ios

Keras introduces a nice way to save and load models.
# serialize model to JSON
model_json = model.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
model.save_weights("model.h5")
print("Saved model to disk")

# later...

# load json and create model
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights("model.h5")
print("Loaded model from disk")

Would be awesome of the keras models could be saved/converted to apple .mlmodel format ready to be impored into ios projects.
See: https://github.com/tensorflow/tensorflow/issues/10468

You can achieve the same result with: https://pypi.python.org/pypi/coremltools


Create a contact us form.

A contact form design that matches the Toggle theme and is responsive.

Assignee: Please assignee this issue to me i am looking to make impactful contribution to the website.

please assign this issue to me

without any company information , call us , and address fields it would be fine just the form and a good UI to the next of it (image).

but what i think so is it is duplicate , check the issue one again before working on it if so pls close this one

ok @abhay-raj19, I have gone through the issues and pull requests. No successful pull requests have been made... That why going to work on this page.


Define EITHER

This PR introduces a new runtime function, (either x y), which returns (if (is? x) x y).
My recent change to reduce was a little awkward. It's strange for reduce to take an argument to be returned if the input list is empty. This PR reverts reduce to the previous version, while using (either (reduce ... in the appropriate places to maintain the existing fixes.
either is also useful in situations where you'd want to use (if (is? x) x y). For example:
(define foo (x)
  (set x (either x 42))
  ...)

I like it. Want to rebase?

Will do!

Did that work? I used Github's "resolve conflicts" interface.


[Pristine] Skip the bundler gem

Since Source::Rubygems will never re-install it
Closes #5578

🎉 nice!

@bundlerbot r+


The recaptcha is not reset if the request returns an error (recaptcha v2 invisible)

Hello, when we send the form for the first time, if the answer is an error then we will go directly to the catch and we will never execute the reset.

It should be indicated in the documentation because I just lost 2 hours just because I am used to make my requests like that:

+1


Docker security

Hi, i have deployed nginx-ui on Docker like documentation but i have a problem: if I try to access to nginx-ui through nginx proxy all is ok (basic auth ask me login) but if i go direct on port 8080 then i can access to nginx-ui bypassing nginx proxy.
So, this is a summary:
nginx.mysite.com -> asking login (OK!!!)
mysite.com:8080 -> i can access to nginx-ui without login.
Any helps?
NGINX config
server {
 listen 80;
 server_name nginx.mysite.com;
 return 301 https://$host$request_uri;
}
server {
    listen 443 ssl;
    server_name nginx.mysite.com;
    ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    location / {
        proxy_pass http://<IP_ADDRESS>:8080/;
    }

    auth_basic "nginxui secured";
    auth_basic_user_file /etc/apache2/.htpasswd;
}

So, if you expose the nginx only on port 443, it wont be able to use port 8080.
If you only use this on the lan, then auth is almost obsolete


ui: experimental -> alpha OP long toggle

I find these two toggles to be a little confusing. You can't enable "Experimental Mode" without having "Experimental openpilot Longitudinal Control" enabled, so the order should be flipped. Both toggles include the word experimental which might be confusing to new users.
Experimental Mode
Experimental openpilot Longitudinal Control

to
openpilot Longitudinal Control
Experimental Mode

Thanks for the PR! I reverted the change to the ordering to ensure the expanded version of the Experimental mode toggle shows without having to scroll.


Clock tree is not balanced because of clock gating

Describe the bug
I don't think this is a bug, it is just a limitation.
The issue is:
I am implementing a design that has part of the clock tree gated. My issue is that in CTS, I can see that there are two clock trees, one for the master clock and one for the gated clock. I can see that in the log too:
[INFO CTS-0010]  Clock net "mprj.CLK" has 10518 sinks.
[INFO CTS-0010]  Clock net "mprj.clk_gated" has 1641 sinks.

The issue is that any path from a FF in one of the clock trees to a FF in the other clock tree has a large skew hence setup and hold violations.
The clock gating is done using a single latch cell. When I remove this cell, the skew become minimal and everything is fine.
So, is there a way that OpenROAD could ignore this cell in CTS and just add it to its leaf so that we have 1 clock tree that is balanced?
I tried adding -balance_levels to the  clock_tree_synthesis but it didn't get balanced.
Expected Behavior
Balanced clock trees across all sinks I guess.
OpenROAD Environment
OpenROAD version: 0a584d123190322b0725d5440c2c486d91d3afd8

OpenLane Environment
n/a

To Reproduce
reproducible.zip.partaa.zip
reproducible.zip.partab.zip
Relevant log output
No response
Screenshots
No response
Additional Context
No response

balance_levels should ensure there are the same number of buffers in all paths but it doesn't ensure that the delays match.  Do you at least see that much?

@maliberty I can see it did ensure that all paths have the same number of buffers, yes. But the skew is still more than 2ns.
Now I looked at 1 of the violating timing paths before and after adding clock gating:
Without clock gating:
The launching flipflop clock had 11 buffers, 3.0 ns delay
The receiving flipflop clock had 10 buffers, 2.2 ns delay
the skew was 0.8 ns and I didn't use balance_levels as the resizer did fix all hold and setup violations.
With the clock gating latch:
The launching flipflop clock had 14 buffers + latch, 5.0 ns delay
The receiving flipflop clock  had 14 buffers, 2.9 ns delay
the skew now is: 2.1 ns and I used balance_levels to add some delay for the receiving flipflop clock.
balance_levels did help but the the delay wasn't matched, yes.

Reporting the skew w/o balance_levels is of no interest as that's known to not work well with clock gating.  Reporting them with balance_levels would be more helpful.

Okay, I will can using balance_levels in both cases. But, is there a solution using OpenROAD for balancing clock trees that has clock gating?

@precisionmoon : " It's kind of urgent for us" from @donn
I also tried adding -obstruction_aware but the results are still pretty bad:

If I manually change mprj.cpu_clk_gate from sky130_fd_sc_hd__dlclkp_1 to sky130_fd_sc_hd__dlclkp_4 then I get:

@donn confirms that the manual workaround resolves the issue for their immediate tapeout.  However we should still fix this when you can.

@maliberty Best status I can gather from a quick download from the team: The Hold timing issue is improved to the point of it being possible to get a netlist by the workaround but at the cost of more than doubling total chip pwr consumption and still requires the addition of wait states to read the flash due to the router stopping working on resolving setup issues. I think we likely need further help here. This is still a significant issues for us,

This is being discussed in slack so I'll respond there


Documentation: Custom Endpoint updates

Environment

Telemetry Streaming Version: 1.12.0
BIG-IP Version: <IP_ADDRESS>.0.11

Summary
While working with example of Custom Endpoints (https://clouddocs.f5.com/products/extensions/f5-telemetry-streaming/latest/custom-endpoints.html), I found the documentation for how to retrieve values from the custom endpoints unclear.  For the example on how to retrieve the data, it has the following:
GET /mgmt/shared/telemetry/systempoller/Custom_System/Custom_System_Poller1
In reality, you need to do a GET request to https://<management_ip_address>//mgmt/shared/telemetry/systempoller/Custom_System/Custom_System_Poller1 in order to be able to retrieve the data.
All of the examples should be updated to make this more clear.
Steps To Reproduce
Documentation issue - no steps to reproduce; just need to make it more clear.

The documentation here is unclear about the fact that it is using a debug feature to confirm the data gathered by custom endpoints. I have added AUTOTOOL-1698 to clarify the custom endpoint documentation by including additional information about this debug functionality.

The documentation has been updated. Closing this issue.


supersim demo

working ecosystem bridge-app locally with supersim. Both forked/vanilla & deposits working from L1

Hi @hamdiallam can you share a link to the demo recording here? Then I think we can close this issue. Thanks!

I think you can close I remember seeing the demo :)


Rework consteval to be more memory efficient.

Also adds an option to use the llvm-cpu backend instead of VMVX. This isn't quite ready for primetime because I need to isolate some flags better, but it can be used in a pinch.
Changes the way that JIT programs are constructed so that stateless functions are compiled at once for all constants and the data is passed in/out vs being cloned from module globals.
Each global has a public function to compute it, allowing us to run them one by one or in batches to keep memory use under control.
In this patch, I am just running them one by one and also haven't optimized memory by use of resources yet. Still a big improvement.
Drops compilation time of h2ogpt from ~45m to 2m47s. More optimization is possible.

JIT'ing globals now takes 52s for this model (vs ~42m with the Linalg const evaler).
Memory pressure is kept under control and does not regress from the current state (more optimization is possible - just a starting point).

I also changed the behavior of the constexpr hoisting and jit passes to produce/process initializers with the iree.compiler.consteval attribute. This was advisable because in the new way of doing evaluation, we are doing a non-general transformation on the initializers and I didn't want it stumbling over arbitrary initializers in the wild. Having the behavior be opt-in seemed prudent.
Flag changes:

iree-consteval-jit-use-vmvx (default true): Uses the VMVX backend. When false, uses the LLVMCPU backend. I'll be doing work to change this to false by default when LLVMCPU is available.
iree-consteval-jit-debug: Prints debugging information about constant evaluation.
iree-opt-const-eval: Flipped to true since it now only processes initializers targeted at it and is therefore safe to always have enabled.

PTAL


warnings during maven build regarding missing version for maven-compiler-plugin

The following warnings show up during a maven build
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.jsmart:http-testing:jar:1.3.36-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 50, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.jsmart:kafka-testing:jar:1.3.36-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 56, column 12
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.jsmart:zerocode-tdd-jupiter:jar:1.3.36-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-compiler-plugin is missing. @ line 145, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]

You can assign this to me for a fix.

I would suggest to create a  section in the parent pom and remove redundant configuration for the maven-compiler-plugin in the child poms

You can assign this to me for a fix.

Assigned now. Thanks for trying to fix these.
When you're ready, please raise a PR and link to this issue.

I have create the pull request https://github.com/authorjapps/zerocode/pull/597
In the parent pom I introduced a  section with configuration for maven-compiler-plugin.
If you agree I would also move the configuration for the other maven-plugins into this  section and update their versions to the most current ones. If these changes are in your sense, please tell me, if I should add these changes to the current pull request or if I should create a new issue and a separate pull request.

Hello @nirmalchandra, is the pull request https://github.com/authorjapps/zerocode/pull/597 acceptable for you?
If you agree I would also move the configuration for the other maven-plugins into this pluginManagement section and update their versions to the most current ones. If these changes are in your sense, please tell me, if I should add these changes to the current pull request or if I should create a new issue and a separate pull request.

Will be done with issue #607 and PR #609


Model downloading speed is too slow and failing.

https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased.tar.gz not found in cache downloading
0%|▏                                                                                | 555008/407873900 [00:29<33:48:58, 3345.85B/s]
then it stops after 1% or 2%

Not a problem related to bertsum. Maybe you want to check https://github.com/huggingface/pytorch-pretrained-BERT or your internet for solutions.


make completion closures @escaping

This PR adds @escaping annotation to the AsyncX completion closures cause Swift made nonescape  closures default(https://github.com/apple/swift/pull/4052).
Almost async functions need to call completion in nested closures that are made with file system, netwroking operations etc...

Yeah, making noescape by default introduced us to a lot of @escaping of boilerplate… Anyway, 👍 for this PR.

We need to add a lot of @escaping boilerplate.. Especially it brought terror to the projects that depend on libuv haha.
@open-swift/contributors Could I merge this?

@noppoMan :shipit:

:shipit:


Fix: add tagbot_onboarding and tagbot_onboarding_started columns and update status at the end of the job

LGTM, I think there is another PR on another repo as well so not merging yet?

I think we can already merge as those changes won't change the current workload.
The other PR is here: https://github.com/trackit/trackit-k8s-dispatcher/pull/3


OSSL_CMP_CTX: rename field and its getter/setter from 'untrusted_certs' to 'untrusted'

This just does some renaming for API consistency and name brevity.

Since also this PR constitutes an API change (though a trivial one)
I'd appreciate a swift review such that this can be merged before feature freeze.

As is visible one is "word_word"(1) and another one is "wordWord" (2) :

@petrovr you are right there is still some inconsistency in the naming of various CMP-related functions.
Generally OpenSSL dues not use CamelCase but names_with_underscores.
For the CMP+CRMF implementation we've just CamelCase for various functions because type and field names such as CertReqMessages and extraCerts are like that in the respective standards (RFC 4210 and RFC 4211).
Yet there are some function names, in particular the trustedStore where the use of CamelCase had some unclear other reasons, and these should better be changed as well for consistency with the OpenSSL naming conventions.

24 hours has passed since 'approval: done' was set, but as this PR has been updated in that time the label 'approval: ready to merge' is not being automatically set.  Please review the updates and set the label manually.

Solved merge conflicts due to changed field name and merged - thanks!


Call to a member function getSourceName() on a non-object in Compiler.php (Line 886)

I apologize if this has been asked before, and I especially apologize if this is an implementation issue on my part. I searched around but only found one (seemingly unrelated) issue.
I'm getting the following fatal error when attempting to use $scss->setLineNumberStyle(\Leafo\ScssPhp\Compiler::LINE_COMMENTS);
Fatal error: Call to a member function getSourceName() on a non-object in .../scssphp/src/Compiler.php on line 886
I'm using that when a debug query string is active, and it only errors on certain .scss files (but I can't find anything common between them that the other successful files don't have).
It was happening with v0.3.4, so I updated to the latest (v0.5.1) with no improvement.
I am including scssphp slightly differently than I've seen others use it, but here is the core of my code:
require_once(get_template_directory() . '/includes/libs/scssphp/scss.inc.php');
$scss = new \Leafo\ScssPhp\Compiler();
$scss->addImportPath(get_template_directory() . '/stylesheets/scss/partials/');
$scss->setFormatter('Leafo\ScssPhp\Formatter\Compact');
$scss->setLineNumberStyle(\Leafo\ScssPhp\Compiler::LINE_COMMENTS); //If this line is commented out, the script works 100%. With it in it works on some files, but not all.
$this_scss_contents = file_get_contents($file); //$file is the file path of the .scss file
$compiled_css = $scss->compile($this_scss_contents);

Here is the entire code implementation for reference.
If it does turn out to be a simple fix, hopefully this will be helpful for anyone else with a similar issue. Regardless, thanks for such an awesome library!

Please see if e9f44a5 fixes this. There was some inconsistency with wrapped blocks.

Thanks for the quick response- unfortunately the error is still occurring. Same fatal error, but the line number moved to 879:
Fatal error: Call to a member function getSourceName() on a non-object in .../scssphp/src/Compiler.php on line 879

Your approach looks ok. Unusual but not unexpected is where nebula_render_scss calls the compile() method multiple times on a single Compiler instance (inside a loop). I don't have an explicit test case for this, so I'll start my investigation there.

Ok. I'm able to reproduce this.

Confirming that this works on my server too. Thanks again for looking into this.

If you get a chance, I'd appreciate it if you'd confirm it still works after 1f5c1eeba750f09653432511876613eba3dd2242

Just tested the latest commit and it's still working great.


YubiKey plugin

Discuss this draft on the age-dev mailing list thread!
Current draft specification:
https://hackmd.io/@str4d/age-plugin-yubikey

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@05f7f1b). Click here to learn what that means.
The diff coverage is 3.57%.

@@          Coverage Diff           @@
##             main      #1   +/-   ##
======================================
  Coverage        ?   3.57%
======================================
  Files           ?       5
  Lines           ?     280
  Branches        ?       0
======================================
  Hits            ?      10
  Misses          ?     270
  Partials        ?       0

Impacted Files
Coverage Δ

src/format.rs
0.00% <0.00%> (ø)

src/main.rs
0.00% <0.00%> (ø)

src/p256.rs
0.00% <0.00%> (ø)

src/plugin.rs
0.00% <0.00%> (ø)

src/yubikey.rs
8.69% <8.69%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05f7f1b...c8389d1. Read the comment docs.

Adding my 5 cents: I would not call it "A YubiKey stanza", but instead "P-256" and not even tie it to PIV (= smartcard stuff), because all of the operations are:

something that can be done in another manufacturer's PIV implementation
or even in a software smartcard implementation
or even outside of PIV context. It's just P-256
the plugin itself is fine being scoped to YubiKey-stored P-256. I'd just decouple the Age stanza and YubiKey from each other.

where SEC-1-C is the 33-byte compressed SEC-1 encoding.

FYI in case you're not aware, if "compressed encoding" means omitting the y coordinate, it needs to be re-computed from x and it seems to be pretty expensive: https://twitter.com/joonas_fi/status/1318829182743359489

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@05f7f1b). Click here to learn what that means.
The diff coverage is 2.89%.

@@          Coverage Diff           @@
##             main      #1   +/-   ##
======================================
  Coverage        ?   2.89%
======================================
  Files           ?       5
  Lines           ?     276
  Branches        ?       0
======================================
  Hits            ?       8
  Misses          ?     268
  Partials        ?       0

Impacted Files
Coverage Δ

src/format.rs
0.00% <0.00%> (ø)

src/main.rs
0.00% <0.00%> (ø)

src/p256.rs
0.00% <0.00%> (ø)

src/plugin.rs
0.00% <0.00%> (ø)

src/yubikey.rs
7.01% <7.01%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 05f7f1b...3d44b5b. Read the comment docs.

Now that (unstable) plugin support is merged into rage (https://github.com/str4d/rage/pull/99), I'll be working on this PR again!
@joonas-fi The stanza tag used in the header is going to end up named something like piv-p256, since (as you note) it's not tied to YubiKey specifically (though it is tied to a specific
The cost of decompression only needs to be paid at recipient parsing time, and we gain more benefit from having shorter recipient strings. I think the current implementation might end up decompressing once per wrapped stanza, but that's easy to fix if it actually proves to be a bottleneck (vs the actual file encryption).

I tested this with YubiKey 4 on Mac and it's working very nicely! The only issue I found is I need to enter PIN on every decryption even though the PIN policy has been set to "once" (?).
I opened a PR (#3) to address a use case that's important to me (first mentioned here: https://github.com/str4d/rage/pull/25#issuecomment-577264766).

I tested this with YubiKey 4 on Mac and it's working very nicely! The only issue I found is I need to enter PIN on every decryption even though the PIN policy has been set to "once" (?).
I opened a PR (#3) to address a use case that's important to me (first mentioned here: https://github.com/str4d/rage/pull/25#issuecomment-577264766).

Hi, thanks for testing! Just FYI, I reworked this branch on a recent Twitch stream, and will shortly be opening new PRs with the improved plugin implementation.
Re: always needing a PIN, that's an artifact of plugins being started fresh each time rage runs. This is intentional; plugins are short-lived. The reason I offer a Once policy is because it will be used by an agent plugin. The expectation is that yubikey-agent will add support for this recipient type, and then this plugin will be used for encryption, while the agent can be used for decryption if the user desires.

Hi, thanks for testing! Just FYI, I reworked this branch on a recent Twitch stream, and will shortly be opening new PRs with the improved plugin implementation.
Re: always needing a PIN, that's an artifact of plugins being started fresh each time rage runs. This is intentional; plugins are short-lived. The reason I offer a Once policy is because it will be used by an agent plugin. The expectation is that yubikey-agent will add support for this recipient type, and then this plugin will be used for encryption, while the agent can be used for decryption if the user desires.

Cool, I'll update my branch when you post your changes. Fair enough on the PIN prompts, I somehow thought the device kept track of PIN entries and it wasn't per-session.

Cool, I'll update my branch when you post your changes. Fair enough on the PIN prompts, I somehow thought the device kept track of PIN entries and it wasn't per-session.

PIN entries are only cached per-connection to the YubiKey, so decrypting multiple files with the same YubiKey would leverage a Once policy correctly (not that age or rage support that use-case, but the plugin system does), but you need some process holding that connection open (and in doing so, holding a lock on the YubiKey) if you want it to last beyond a single plugin invocation.

PIN entries are only cached per-connection to the YubiKey, so decrypting multiple files with the same YubiKey would leverage a Once policy correctly (not that age or rage support that use-case, but the plugin system does), but you need some process holding that connection open (and in doing so, holding a lock on the YubiKey) if you want it to last beyond a single plugin invocation.

Thanks, that's what I figured. You mentioned yubikey-agent, is it about this one? https://github.com/FiloSottile/yubikey-agent If there was a single agent that'd serve both my SSH and (r)age needs it'd be glorious.

Thanks, that's what I figured. You mentioned yubikey-agent, is it about this one? https://github.com/FiloSottile/yubikey-agent If there was a single agent that'd serve both my SSH and (r)age needs it'd be glorious.

The expectation is that yubikey-agent will add support for this recipient type, and then this plugin will be used for encryption, while the agent can be used for decryption if the user desires.

One more question about this: would this reuse the SSH agent protocol and extend it to support decryption or provide a separate protocol for (r)age? Is there an existing discussion of this yubikey-agent extension? I've played around with sshcrypt[1] and yubikey-agent (a version modified by me to get deterministic RSA signatures working instead of ECDSA signatures which are randomized) to get encryption and decryption using SSH keys with agent support but it has a range of downsides (like: encryption requires SSH sign operation with a touch action and requiring signature to encrypt means a third party can't encrypt stuff to me). I had an idea for an age plugin that implements something like that using SSH agent but it'd have the same set of downsides which is a showstopper for me.
[1] https://github.com/leighmcculloch/sshcrypt

The expectation is that yubikey-agent will add support for this recipient type, and then this plugin will be used for encryption, while the agent can be used for decryption if the user desires.

One more question about this: would this reuse the SSH agent protocol and extend it to support decryption or provide a separate protocol for (r)age? Is there an existing discussion of this yubikey-agent extension? I've played around with sshcrypt[1] and yubikey-agent (a version modified by me to get deterministic RSA signatures working instead of ECDSA signatures which are randomized) to get encryption and decryption using SSH keys with agent support but it has a range of downsides (like: encryption requires SSH sign operation with a touch action and requiring signature to encrypt means a third party can't encrypt stuff to me). I had an idea for an age plugin that implements something like that using SSH agent but it'd have the same set of downsides which is a showstopper for me.
[1] https://github.com/leighmcculloch/sshcrypt

Replaced by #8.

One more question about this: would this reuse the SSH agent protocol and extend it to support decryption or provide a separate protocol for (r)age? Is there an existing discussion of this yubikey-agent extension?

The existing discussion is mostly in @FiloSottile's Twitch streams IIRC, along with he and I chatting about it. I suspect it will be a protocol specific to yubikey-agent, i.e. yubikey-agent would speak the SSH agent protocol for SSH usage, and then have its own protocol between its short-lived plugin binary and its long-lived agent. Let's shift that conversation over to yubikey-agent.


Hotkeys disabled when control buttons focused

@ctd1500
This particular line disables hotkeys: https://github.com/ctd1500/videojs-hotkeys/blob/master/videojs.hotkeys.js#L71
The comment above this section suggests an option to disable this behaviour, but there is no option available.
The comment suggests disabling is possible if in jogStyle mode, but this does not seem like the correct name for such a control. Perhaps a new option is required?
One particular example:
User clicks fullscreen button, then presses spacebar to pause.
Expected behaviour:
Video goes fullscreen, then is paused.
Actual behaviour:
Video goes fullscreen, then exits fullscreen.
I am able to submit a PR which fixes the issue if required.

It acts this way so as to not override accessibility options (tabbing between buttons, and hitting space to "click" them).
But if you'd like, you can submit a PR to add an option to disable that.

In #18 I've made the option default to false, but I would argue that because ENTER/RETURN can be used instead of SPACE for accessibility, that the option should perhaps default to true from 0.3 onward.


[Feature] Add env support in instruct.txt

For example, my prompt is as following
please write a semantic commit message for the diff, releated issue id is ${branchName}

Better to see the value someway in case of typos. At least, $ should trigger a list of env to choose from or press tab to complete one.

In my use case, I have delegated the task of writing commit messages to DevChat. The goal is to provide a complete prompt with just the command /commit_message. In general, we use the issue_id as the branch name. If this information can be included in the instruct.txt, there's no need to input "related Jira issue is $branchName" every time.
Additionally, adding a shortcut to reference environment variables during input would be a useful feature.

This feature has supported in version 0.1.7. for example: ${branchName} will replaced by env var branchName.


adding VPN info for building data

@siunies   this PR is for the live branch, please submit updates to the internal working branch. I've taken these changes and put them in the working branch, closing this PR now.


hotlink protaction

Hello sir
how to protect  rtmp hotlink other site (without nginx securelink)
any easy way ?
i use
play /myvideofolder/

hi arut how are you ? i need some help with securing my nginx rtmp server so that stream only play on my website, some sort of restreaming restrictions

@gdulli111: Use HLS with crossdomain.xml and deny CORS.

hi thank you for the reply really appreciate i tried following few
instructions but was unable can you give me some instructions please
kind regards
On 7 March 2016 at 22:20, Andrew Shulgin<EMAIL_ADDRESS>wrote:

@gdulli111 https://github.com/gdulli111: Use HLS with crossdomain.xml
and deny CORS.
—
Reply to this email directly or view it on GitHub
https://github.com/arut/nginx-rtmp-module/issues/229#issuecomment-193481463
.


Query about recomposing markdown or html

Good
I'm new and I'm a little lost.
From what I have seen when you forward a publication to the bot, telegram separates the text from the format in the json sent.
I've been looking but I have not seen any utility to recompose the publication in markdown or html format.
Is there any?
Thank you

Hi @josegarzi2
If by "recompose" you mean to edit an existing message, have a look at the official API:
https://core.telegram.org/bots/api#editmessagetext
You can simply pass along the parse_mode parameter there too, same as sending a new message.

What I am trying is:
Forward a post to the bot.  Telegram sends a json where it separates the format text (bold, italics, ...)
What I want is to convert the text with its format to markdown or html to be able to edit it and modify the publication that was forwarded.
I do not know if there is a helper for this

You will need to write yourself a parser that puts back the formating entities on their place.
This can be a bit buggy eot multibyte characters so watch out.

thanks


[csl] states and scheduling simplification

After enabling the scheduling of delayed reception slots in #7677
they would be configured even when the device switched to MED mode.
This commit introduces some simplifications around the CSL states
and scheduling:

Remove kStateCslSample Sub-Mac state.
Mantain the CSL timer stopped if not in CSL Receiver mode.
Keep the CSL Period value in the MAC in order to use it as an
indication of CSL Receiver mode in the Sub-Mac.

v1_2_test_csl_transmission.py still failing after setting the CSL Period to 0. Need to figure out why.

@abtink @Irving-cl I made some changes to recover kStateCslSample. Could you take a look?


[bug] [macOS] [Updater]: v1Compatible does not create updater artifacts for macOS

Describe the bug
Setting
"createUpdaterArtifacts": "v1Compatible",
does not create v1 compatible updater artifacts or signature files for macOS. Only for Windows.
Reproduction
No response
Expected behavior
No response
Full tauri info output
❯ cargo tauri info
    Error `tauri.conf.json` error on `bundle`: Additional properties are not allowed ('createUpdaterArtifacts' was unexpected)

Stack trace
No response
Additional context
No response

Sounds like your tauri cli version is outdated.
Does it really complain about that when you run cargo tauri info? It's supposed to run regardless of tauri.conf.json errors :(
Anyway, you can check the cli version with cargo tauri --version, just make sure it's 2.0.0-beta.21

Ah, thanks for the suggestion
Here's my tauri info.
After updating and running the build again. It's still not generating v1 compatible updater artifacts for macOS.
I also updated tauri-plugin-updater to version 2.0.0-beta.9
    - OS: Mac OS 14.4.0 X64
    ✔ Xcode Command Line Tools: installed
    ✔ rustc: 1.77.2 (25ef9e3d8 2024-04-09)
    ✔ cargo: 1.77.2 (e52e36006 2024-03-26)
    ✔ rustup: 1.27.0 (bbb9276d2 2024-03-08)
    ✔ Rust toolchain: stable-aarch64-apple-darwin (environment override by RUSTUP_TOOLCHAIN)
    - node: 21.6.1
    - pnpm: 8.15.2
    - npm: 10.2.4
    - bun: 1.0.26

[-] Packages
    - tauri [RUST]: 2.0.0-beta.23
    - tauri-build [RUST]: 2.0.0-beta.18
    - wry [RUST]: 0.41.0
    - tao [RUST]: 0.28.1
    - tauri-cli [RUST]: 2.0.0-beta.21
    - @tauri-apps/api [NPM]: 2.0.0-beta.14
    - @tauri-apps/cli [NPM]: 2.0.0-beta.21

[-] App
    - build-type: bundle
    - CSP: unset
    - frontendDist: ../dist
    - devUrl: http://localhost:1420/
    - framework: React
    - bundler: Vite

What's your bundle.targets config in tauri.conf.json? It now requires "app" (we messed up the condition for the warning, but that's fixed for the next release).

Found the issue. I needed to change the target from dmg to app

What's your bundle.targets config in tauri.conf.json? It now requires "app" (we messed up the condition for the warning, but that's fixed for the next release).

Found the issue, I need to change the target to app from dmg. Thanks!!

What's your bundle.targets config in tauri.conf.json? It now requires "app" in the array (if you don't have it set to "all"). We messed up the condition for the warning, but that's fixed for the next release.

Saw this warning in my build as soon as posted a reply

Found the issue, I need to change the target to app from dmg. Thanks!!

Btw, you can have both enabled (does not apply to linux/windows). dmg is just a wrapper around the .app bundle so if you install your app via dmg it can be updated with the .app bundle just fine.


Stop tip footer only for prefix interactions

Just made it check whether an interaction had a user property and if so, to leave out the stop tip footer since there is a stop button that they can click. I tested it and it seems to be working fine.

You have solved the problem... but in the wrong way :/
Even if useButtons is set to false and the user passes in an interaction, they should in theory still be able to send "S" or "Stop" to end their game.
I'll change it so that the footer property only appears if useButtons has been set to true.

Oops, my bad, I totally forgot that people could select whether they wanted to use buttons or not. Glad to see that you released your own patch, though.


Raiden Smart Contract Limits

For the Red Eyes release we want to put artificial limits on the smart contracts for security reasons.

[ ] Figure out the amount of limits that should be put and how they should be. Kind of difficult due to different value of each token. For now let's go with 100 tokens.

[ ] Implement the limits in the contracts

[ ] Add some tests for the contract limits

I made this as little helper sheet to judge the impact of the limit: https://docs.google.com/spreadsheets/d/1zybi5HpElVBoMTmYLMFwNv1T4VjCclHaiCsm6Li9atI/edit#gid=0
Questions:

Is USD 50k low enough for the upper barrier?
Does USD 10 make sense as a "cut-off" point where below that you would implicitly "exclude" these tokens for practical use?

If we can answer both of these with yes, I'd argue that (given the sheet and the tokens we'd "exclude") that the 100 tokens limit wouldn't be too bad. If anything, I'd say I'd say it could even be 50 tokens and we wouldn't exclude too many "important" tokens.

As all tokens of one token network are in the same contract, the limit per channel does not limit the total amount that is managed by the contracts. Is this intentional?
Related: Is there a plan what happens if there is a bug and tokens are lost?

Duplicate with https://github.com/raiden-network/team/issues/54


Proposal: Loosen Import Redirection (1809+) restrictions for Desktop Application use

Proposal: Loosen Import Redirection (1809+) restrictions for Desktop Application use
Summary
Import Redirection, a feature introduced in Windows 10 version 1809, enables packaged application developers to "redirect" API imports to other implementations (whether they be private or elsewhere in the OS) without modifications to the application, its dependencies, or any related source code.
Use of this feature is currently restricted to packaged applications. This proposal seeks to loosen this restriction and allow this feature to be used with desktop applications.
Rationale
Allowing unpackaged desktop applications to utilize Import Redirection has several benefits, for example:

Puts the responsibility of loading redirection DLLs and the act of redirecting APIs onto the Windows Loader, eliminating the need for injectors and/or loaders.

Provides a simple (and documented) model for redirecting APIs, bringing a level of order to chaos.

Real world example: Microsoft Edge / Chromium use runtime import table patching to redirect APIs at runtime, to, for example, handle the lack of gdi32, user32, and other modules not present on Windows 10X. Presumably, all this code could be replaced with a Import Redirection DLL.

Shortens the hook development/test loop considerably as all the hooking logic is automatically handled by Windows.

Aligns the features more closely with LD_PRELOAD, a popular feature used on other platforms.

Scope

Capability
Priority

Desktop application can be configured to use an Import Redirection DLL without modification to its files or dependencies on disk
Must

Desktop application Import Redirection can redirect all APIs without restriction
Must

Desktop application Import Redirection works without administrative privileges
Should

Desktop application Import Redirection mark processes as redirected for easy identification
Should

Desktop application Import Redirection avoids using the Windows Registry as a mechanism to configure this behavior
Should

Desktop application Import Redirection is tied to a Limited Access Feature
Could

Desktop application Import Redirection redirects APIs with module-level granularity
Could

Desktop application Import Redirection DLLs, marked as optional, are skipped if failure occurs
Could

Desktop application Import Redirection supports multiple DLLs and applies redirects in configured order
Could

Desktop application Import Redirection DLL signature is checked for validity
Could

Desktop application Import Redirection supports explicit specification of DLL architecture
Could

Desktop application Import Redirection ignores, by default, applications signed by Microsoft
Could

Desktop application Import Redirection DLL requires packaging/installation
Won't

Desktop application Import Redirection uses DLL Search Order or PATH to locate its configuration
Won't

Important Notes
Recommend syncing with App Lifecycle proposal due to its desktop application manifestation proposal. If implemented, the aforementioned desktop application manfiest could be extended to contain Import Redirection configuration parameters as well.
Sample usage

User creates a msix manifest for a desktop app (fabrikam.exe)
<?xml version="1.0" encoding="utf-8"?>
  <Identity Name="FabrikamDesktopApplication" ... />
  <ImportRedirectionTables architecture="x86">
    <Module name="app.exe"> <!-- Note: Module-level granularity is a Could feature -->
      <File>ContosoRedirections.dll</File>
      <File>NorthwindRedirections.dll</File>
    </Module>
    <Module name="dependency.dll">
      <!-- Note: Optional flag is a Could feature -->
      <File Optional="true">ADatumRedirections.dll</File>
    </Module>
  </ImportRedirectionTables>

This manifest is saved to disk as fabrikam.exe.msixmanifest, located side-by-side with fabrikam.exe.

Thanks @riverar . Import Redirection was designed to be applicable for Win32 apps as well; the original implementation was scoped to MSIX (and focused on UWP) as a prioritization / simplification choice, but there is nothing technically stopping it working for other apps. Finding a "home" for the redirection metadata is the biggest missing piece.
One caveat: API Redirection does not work for everything; DLLs loaded early in process startup (like NTDLL) cannot be redirected since they are resolved before the redirection tables are loaded. And as currently implemented, redirection does not apply to system binaries; only binaries inside the app's package graph are subject to having their imports redirected. Without a package to consult, we'd need a way to explicitly specify which DLLs to redirect (possibly with wild-cards etc.).
Initial response to your questions:

Does the specification of redirection DLLs create any new security concerns?

For packaged apps (the current scenario) the answer is "no," since you can't tamper with the manifest or any of the DLLs without breaking the signature. For unpackaged apps, it's a new feature that would need a thorough threat analysis but my initial (brief) thinking is that it's no worse than existing attacks against unpackaged apps (exploiting weak ACLs, etc.).

Should redirection DLLs be signed?

Should they? Yes, since all DLLs should be signed :-). See below for related questions...

Should the application fail to launch if redirection DLLs are specified but cannot be found

Right now it's considered fatal because it's part of a package where presumably you're in complete control of everything and the redirection is assumed to be mandatory (there are no decent diagnostics for this failure, which something we should improve...). For unpackaged apps, where you might not be in control of everything (eg, due to plugins, different SKU options, etc.) it might be reasonable to have an "optional" flag in the manifest. Is this necessary for a "v1" version, or is it something we could add later if we see the need?

Should Import Redirection support Windows on ARM (CHPE) scenarios?

Is there a reason why you call this out specifically? In a vacuum, the answer should be "yes."

Should Import Redirection work for .NET applications?

It does today for .NET Native (since that uses IATs) but not for .NET Core (or CoreCLR, used for UWP debug builds) which rely on LoadLibrary / GetProcAddress to resolve DllImports. Including LL / GPA in API Redirection is a missing feature, and if we implemented that it would work for .NET apps, too. Perhaps though what you are really asking is "should there be a pure .NET way of doing this instead of having to write a native DLL"?
There are a few other things to think about; we would like the community's feedback on them.

As noted above, API Redirection does not apply to system binaries today. Should it be an option to do so? Note this could cause all sorts of problems (customer takes a Windows Update and now their app crashes because your redirection implementation doesn't work for that use-case) and would be highly undesirable. But do you think it should it be an option?
In addition to implicit redirection ("oh I found this manifest, let me load the DLL") we'd considered explicit redirection as well, e.g. a way to tell CreateProcess "...and here are the DLLs to search for redirection tables." Interesting? High or low priority?
Currently, redirection applies to all app binaries; if you redirect Foo to Bar then every app call to Foo goes to Bar. We had initially considered a per-app-binary redirection, so you could redirect (say) all calls from Foo to Bar when made by mine.dll, but you could redirect Foo to OtherBar (or not redirect at all) for yours.dll. Interesting? High or low priority?
With regards to signing, is it important to have an option to require signing the manifest file? And the redirection DLLs? Signed by whom? (Presumably the person writing the manifest / redirection DLL is not the original app author, so it probably isn't "the same" cert used to sign the app).
More broadly we have heard from customers that they want to require that all DLLs loaded by their process are signed by <set of certificates>. This is not strictly a security feature (an attacker could simply patch the main EXE to turn off the "I require signing" bits) but is more about reliability; some apps don't want to "accidentally" load random DLLs, or have 3rd-party hooks injected into themselves, etc. If this is interesting, please consider opening another issue for it.

Should Import Redirection support Windows on ARM (CHPE) scenarios?
Is there a reason why you call this out specifically? In a vacuum, the answer should be "yes."

Just thinking users may want to provide a CHPE Redirection DLL for ARM users. Edge case for sure.

Should Import Redirection work for .NET applications?

It does today for .NET Native (since that uses IATs) but not for .NET Core (or CoreCLR, used for UWP debug builds) which rely on LoadLibrary / GetProcAddress to resolve DllImports. Including LL / GPA in API Redirection is a missing feature, and if we implemented that it would work for .NET apps, too. Perhaps though what you are really asking is "should there be a pure .NET way of doing this instead of having to write a native DLL"?

Actually, was thinking of the former case which you covered 👍 I think hooking .NET is out of scope and can be handled with CLR profiling.

There are a few other things to think about; we would like the community's feedback on them.

As noted above, API Redirection does not apply to system binaries today. Should it be an option to do so? Note this could cause all sorts of problems (customer takes a Windows Update and now their app crashes because your redirection implementation doesn't work for that use-case) and would be highly undesirable. But do you think it should it be an option?

What's a "system binary" here? Are we talking about known DLLs (e.g. kernel32), apps (e.g. svchost), both/neither?

In addition to implicit redirection ("oh I found this manifest, let me load the DLL") we'd considered explicit redirection as well, e.g. a way to tell CreateProcess "...and here are the DLLs to search for redirection tables." Interesting? High or low priority?

Interesting indeed, would need to be wired into ActivateApplication as well. Should it respect the appmodel sandbox, if present?

Currently, redirection applies to all app binaries; if you redirect Foo to Bar then every app call to Foo goes to Bar. We had initially considered a per-app-binary redirection, so you could redirect (say) all calls from Foo to Bar when made by mine.dll, but you could redirect Foo to OtherBar (or not redirect at all) for yours.dll. Interesting? High or low priority?

Hm. It's my understanding users could roll out similar behavior via __ShouldApplyRedirectionToFunction__ (untested). If so, I'm not sure there's much value to be had.

With regards to signing, is it important to have an option to require signing the manifest file? And the redirection DLLs? Signed by whom? (Presumably the person writing the manifest / redirection DLL is not the original app author, so it probably isn't "the same" cert used to sign the app).

Hm, on second pass, I don't see a lot of value to be had with signing. Perhaps we toss that out.

More broadly we have heard from customers that they want to require that all DLLs loaded by their process are signed by <set of certificates>. This is not strictly a security feature (an attacker could simply patch the main EXE to turn off the "I require signing" bits) but is more about reliability; some apps don't want to "accidentally" load random DLLs, or have 3rd-party hooks injected into themselves, etc. If this is interesting, please consider opening another issue for it.

I don't think that'll ever truly work. Microsoft (e.g. PowerToys), NVIDIA (e.g. nvapi), and others inject all sorts of hooks into processes on the system for all sorts of reasons.

What's a "system binary" here? Are we talking about known DLLs (e.g. kernel32), apps (e.g. svchost), both/neither?

In the current context, "system binary" meant anything not in the package. For unpackaged apps, since nothing is in a package :) the definition would need to be updated (eg, maybe everything in %system32%, or anything not in the app's directory & subdirectories, or some specific list of do-not-virtualize DLLs, etc.). At a minimum we likely need a speedbump to prevent accidental redirection of DLLs that could app instability or crashes. Have you ever needed to redirect an API for a DLL that wasn't "part of the app"?

Should it respect the appmodel sandbox, if present?

Redirection doesn't / can't affect the sandbox. It's basically just switching a pointer.

It's my understanding users could roll out similar behavior via the existing ShouldApplyRedirectionToFunction (untested)

That wouldn't let you do a different remapping though. (I was going to say "that's not documented" but then again none of this is very well documented...).

Microsoft (e.g. PowerToys), NVIDIA (e.g. nvapi), and others inject all sorts of hooks into processes on the system for all sorts of valid reasons.

Anyone who opts into this likely knows what they are doing and will test appropriately; whatever PowerToys is trying to inject isn't that essential (sorry Clint :)). Anyway, good to know you're not specifically interested in it.

What's a "system binary" here? Are we talking about known DLLs (e.g. kernel32), apps (e.g. svchost), both/neither?
In the current context, "system binary" meant anything not in the package. For unpackaged apps, since nothing is in a package :) the definition would need to be updated (eg, maybe everything in %system32%, or anything not in the app's directory & subdirectories, or some specific list of do-not-virtualize DLLs, etc.). At a minimum we likely need a speedbump to prevent accidental redirection of DLLs that could app instability or crashes. Have you ever needed to redirect an API for a DLL that wasn't "part of the app"?

I'm still struggling with the write up on this point so I'll try to restate it and we'll bang away on that.

Import Redirection must allow a user to redirect any/all APIs listed in the IAT of any manifested executable, no restrictions. Hooking APIs that would normally call an operating system DLLs is critical to most appcompat work. (For example, for Myst, I had to fix up DirectInput!EnumDevices.)

Import Redirection (metadata) could be ignored for system executables, with system executables defined as executables owned (and signed?) by Microsoft (?). This feels especially rigid and error prone, am eager to hear your take on this.

It's my understanding users could roll out similar behavior via the existing ShouldApplyRedirectionToFunction (untested)
That wouldn't let you do a different remapping though. (I was going to say "that's not documented" but then again none of this is very well documented...).

If I understand correctly, the question is whether or not users want to redirect APIs at module-granularity or if process-granularity is sufficient. That's an interesting question.
So presumably, the metadata could appear as such:
<?xml version="1.0" encoding="utf-8"?>
  <Identity Name="DesktopApplication1" ... />
  <ImportRedirectionTables architecture="x86">
    <Module name="app.exe">
      <File>ContosoAppRedirections.dll</File>
      <File>NorthwindAppRedirections.dll</File>
    </Module>
    <Module name="dependency.dll">
      <File>ADatumAppRedirections.dll</File>
    </Module>
  </ImportRedirectionTables>
  ...

Module-level granularity sounds like a great stretch goal, I'd consider it a candidate for addition to proposal's scope as a Could item.
Administrative question: How/When do we go about revising the proposal as we continue the discussion?

Shameless bump. Can we bang on this some more @ptorr-msft?

@ptorr-msft has limited availability right now. @jonwis, is there anyone else who can engage?

@riverar can you update the original proposal with context from the discussion, including proposed manifest changes (Win32 and MSIX)? Then we can generate internal work-item to track it. I also think you can drop a lot of the "rationale" if you like.

Updated the proposal with what we've discussed so far. Pared down the rationale as well.
Any additional thoughts @ptorr-msft and friends?

Updated the proposal with what we've discussed so far. Pared down the rationale as well.
Any additional thoughts @ptorr-msft and friends?

Ping to renew interest for 2023+

Understand everyone is busy and resources are, well, less than before.
@AdamBraden Is your assignment here still valid? Or can we hot potato this to another person? I still think this feature would be very well received! cc: @jonwis

Thanks for the ping, Rafael. Let me work with @jonwis and others to see if we can find the right owner and find a way to prioritize it.

Just my yearly reminder that it's always a great time to consider making import redirection available to desktop apps.
Completing this feature would look great in your annual reviews too. Here's what you're missing out on:

Enabled enterprise integration across 20+ line-of-business applications, with early testing showing potential productivity gains of up to 40% in target workflows

Advanced digital preservation initiatives by modernizing legacy gaming solutions, improving compatibility for an estimated 100k+ titles and potentially impacting millions of users worldwide

Implemented standardized import hook architecture reducing custom security implementations by an estimated 75%, with initial assessments suggesting up to 60% decrease in related vulnerability reports

Happy holidays. 🎅


migrate Json.Of to JsonOf

@h1alexbel thank you for reporting this. I'll assign someone to take care of it soon.

@h1alexbel this is your task now, please go ahead. Deadline (when this ticket should be closed) is 2023-07-03T14:49:05.561795.
Estimation here is 60 minutes, that's how much you will be paid.
Remember, you don't have to solve everything in this ticket - you can solve it partially and leave todo markers in the code, which will become future tasks.
If you have any questions don't ask me, I'm not a technical person. Open new tickets instead.
If you don't have time or simply don't want to work on this, you can always resign.

@h1alexbel thank you for resolving this ticket. I've just added it to your active invoice. You can always check all your invoices and more on the Contributor Dashboard.


Properly use the config on akka connection pools

This is the same issue as #195.
Sparql Triggered processor uses connection pool that overrides the config. make sure it's done right.

[x] fix the tsv get connection pool in sparql triggered processor
[x] check newHostConnetionPool, newCachedConnetionPool etc. for config overrides generally on whole cm-well
[x] currently it shouldn't work. Why there isn't any error printed in the logs??? (the size issue is fixed in code by withoutSizeLimit)

Fixed by PR #242

@Ivan-Shestakov - please test it


[ERROR]Task Node(3-unzip/reads/dump_rawread_ids) failed with exit-code=1

Hello All,
I am having an error while using falcon_unzip (be installed the pb-assembly), the error is as follows. My genome size is about 3.43Mb.
Thanks a lot for your help.
[INFO](slept_ for another 2.9999999999999996s -- another 4 loop iterations)
[INFO](slept for another 10.500000000000002s -- another 6 loop iterations)
[ERROR]Task Node(3-unzip/reads/dump_rawread_ids) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: {Node(3-unzip/reads/dump_rawread_ids)}
[ERROR]ready: set()
                      submitted: set()
[ERROR]Noop. We cannot kill blocked threads.
[ERROR]Error for run(target=clr, config_fn=fc_unzip.cfg, logging_config_fn=None)

Traceback (most recent call last):
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 195, in run
   run_logged(target, config_fn)
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 184, in run_logged
  unzip_all(config, unzip_config_fn)
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all
   tasks_unzip.run_workflow(wf, config, unzip_config_fn)
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/falcon_unzip/tasks/unzip.py", line 467, in run_workflow
   dist=Dist(NPROC=24, job_dict=config['job.step.unzip.blasr_aln']),
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/falcon_kit/pype.py", line 109, in gen_parallel_tasks
   wf.refreshTargets()
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
  self._refreshTargets(updateFreq, exitOnFailure)
 File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
 raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(3-unzip/reads/dump_rawread_ids)}

My Confif file is as follows:

fc_unzip.cfg

[General]
max_n_open_files = 1000
> [Unzip]
> input_fofn=` input.fofn
> input_bam_fofn= input_bam.fofn
> #sge_phasing= -pe smp 12 -q bigmem
> #sge_quiver= -pe smp 12 -q sequel-farm
> #sge_track_reads= -pe smp 12 -q default
> #sge_blasr_aln=  -pe smp 24 -q bigmem
> #sge_hasm=  -pe smp 48 -q bigmem
> #unzip_concurrent_jobs = 64
> #quiver_concurrent_jobs = 64
>
> #unzip_concurrent_jobs = 12
> #quiver_concurrent_jobs = 12
>
> [job.defaults]
> NPROC=6
> njobs=32
> #job_type = SGE
> job_type = local
>
> #use_tmpdir = /scratch
> pwatcher_type = blocking
> #job_type = string
> submit = bash -C ${CMD} >| ${STDOUT_FILE} 2>| ${STDERR_FILE}
>
> #njobs=120
> #njobs=8
> #NPROC=4
> #submit = qsub -S /bin/bash -sync y -V  \

> [job.step.unzip.track_reads]
> njobs=1
> NPROC=48
> [job.step.unzip.blasr_aln]
> njobs=2
> NPROC=16
> [job.step.unzip.phasing]
> njobs=16
> NPROC=2
> [job.step.unzip.hasm]
> njobs=1
> NPROC=48
> [job.step.unzip.quiver]
> njobs=2
> NPROC=12

### run-P8053c4ab3cf5d1.bash.stderr
> executable=${PYPEFLOW_JOB_START_SCRIPT}
> + executable=/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/run-P8053c4ab3cf5d1.bash
> timeout=${PYPEFLOW_JOB_START_TIMEOUT:-60} # wait 60s by default
> + timeout=60
>

> while [[ ! -x "${executable}" ]]; do
>     if [[ "${timeout}" == "0" ]]; then
>         echo "timed out waiting for (${executable})"
>         exit 77
>     fi
>     echo "not executable: '${executable}', waiting ${timeout}s"
>     sleep 1
>     timeout=$((timeout-1))
> done
> + [[ ! -x /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/run-P8053c4ab3cf5d1.bash ]]
>
> /bin/bash ${executable}
> + /bin/bash /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/run-P8053c4ab3cf5d1.bash
> + '[' '!' -d /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids ']'
> + cd /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids
> + eval '/bin/bash run.sh'
> ++ /bin/bash run.sh
> export PATH=$PATH:/bin
> + export PATH=/root/miniconda3/envs/dna/bin:/root/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/miniconda3/bin:/root/bin:/bin
> + PATH=/root/miniconda3/envs/dna/bin:/root/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/miniconda3/bin:/root/bin:/bin
> cd /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids
> + cd /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids
> /bin/bash task.sh
> + /bin/bash task.sh
> pypeflow 2.3.0
> 2019-10-12 10:16:58,020 - root - DEBUG - Running "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/task.json"
> 2019-10-12 10:16:58,021 - root - DEBUG - Checking existence of '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/task.json' with timeout=30
> 2019-10-12 10:16:58,021 - root - DEBUG - Loading JSON from '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/task.json'
> 2019-10-12 10:16:58,022 - root - DEBUG - {'bash_template_fn': 'template.sh',
>  'inputs': {'falcon_asm_done': '../../../2-asm-falcon/falcon_asm_done',
>             'rawread_db': '../../../0-rawreads/build/raw_reads.db'},
>  'outputs': {'rawread_id_file': 'rawread_ids'},
>  'parameters': {'pypeflow_mb': 4000, 'pypeflow_nproc': 1, 'topdir': '../../..'}}
> 2019-10-12 10:16:58,023 - root - INFO - CD: '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids' <- '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids'
> 2019-10-12 10:16:58,023 - root - DEBUG - Checking existence of '../../../2-asm-falcon/falcon_asm_done' with timeout=30
> 2019-10-12 10:16:58,023 - root - DEBUG - Checking existence of '../../../0-rawreads/build/raw_reads.db' with timeout=30
> 2019-10-12 10:17:28,058 - root - ERROR - Was waiting for '../../../0-rawreads/build/raw_reads.db'

> Traceback (most recent call last):
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 69, in wait_for
>     _wait_for(fn, timeout)
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 85, in _wait_for
>     raise Exception('Timed out waiting for {!r}'.format(fn))
> Exception: Timed out waiting for '../../../0-rawreads/build/raw_reads.db'
> 2019-10-12 10:17:28,059 - root - INFO - CD: '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids' -> '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids'
> 2019-10-12 10:17:28,059 - root - CRITICAL - Error in /root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py with args="{'json_fn': '/public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
> Traceback (most recent call last):
>   File "/root/miniconda3/envs/dna/lib/python3.7/runpy.py", line 193, in _run_module_as_main
>     "__main__", mod_spec)
>   File "/root/miniconda3/envs/dna/lib/python3.7/runpy.py", line 85, in _run_code
>     exec(code, run_globals)
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 280, in <module>
>     main()
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 272, in main
>     run(**vars(parsed_args))
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 266, in run
>     run_cfg_in_tmpdir(cfg, tmpdir, '.')
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 219, in run_cfg_in_tmpdir
>     wait_for(fn)
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 69, in wait_for
>     _wait_for(fn, timeout)
>   File "/root/miniconda3/envs/dna/lib/python3.7/site-packages/pypeflow/do_task.py", line 85, in _wait_for
>     raise Exception('Timed out waiting for {!r}'.format(fn))
> Exception: Timed out waiting for '../../../0-rawreads/build/raw_reads.db'

> +++ pwd
> ++ echo 'FAILURE. Running top in /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
> ++ rm -f top.txt
> ++ which python
> ++ which top
> ++ env -u LD_LIBRARY_PATH top -b -n 1
> ++ env -u LD_LIBRARY_PATH top -b -n 1
> ++ pstree -apl
>
> real	0m30.740s
> user	0m0.320s
> sys	0m0.275s
> + finish
> + echo 'finish code: 1'

Exception: Timed out waiting for '../../../0-rawreads/build/raw_reads.db'

You are missing that file (w.r.t. /public/home/FALCON/DBM/3-unzip/reads/dump_rawread_ids/)


update readme to include instructions for clearing dns cache on Mint

Linux Mint is in the tree of Debian/Ubuntu and so you'd expect that those instructions would work. However Mint has its own configuration. As Mint is one of the most popular distros, probably worth having.

Thanks Dave @mebigfatguy!  That's a perfect PR, right there.
Except for one thing – god I really should document this but I'm pretty sure it won't help.
readme.md is generative.  It's produced from data merged-in at the end of the Python build process.  So this change belongs in readme_template.md.
Dave, can you amend this PR, making the change in readme_template.md?

That's what I'm talkin' about!  Brosius!


[skip ci] Add a Cirrus-ci.com build badge

Reported-by: Alex Zepeda

@bors: r+

:pushpin: Commit e3bb3d7f8e258c88e3d9c3e46154654e61764e1d has been approved by gnzlbg

:hourglass: Testing commit e3bb3d7f8e258c88e3d9c3e46154654e61764e1d with merge b0826af02054a7cb05c59cee5a424cead6f606ba...

:boom: Test timed out

@bors: retry

:hourglass: Testing commit e3bb3d7f8e258c88e3d9c3e46154654e61764e1d with merge 34656d64602e9a78d358ac251263989dd61d4d84...

It's fine to merge this manually, the "skip ci" is causing CI to not run which breaks bors

lol.  Maybe it should be called [break ci]?


Tool comparison documentation

@adrianwebb I know you wish to update the comparison matrix more.  I want to merge because I think I'm confused about what's in what branch.  Let's simplify it, and make a new branch from "update-matrix" or similar.


Head input chain offset

When a participant starts an f3 instance, it needs to determine whether it wants its proposal to contain from baseChain up to the head input chain it locally keeps in EC, or some prefix of it (i.e. offset it by some epochs). Although this may delay latency on the head tipset, it helps reduce latency and reach agreement faster on the EC instance. This is closely related with #173.

In pair with decision on #173 to wait for 2 epochs from the timestamp on the latest finalized epoch, the participant sets as proposal for the new f3 instance the following:

Let $baseChain$ represent the finalized chain up until the latest finalized f3 instance.
Let $ECChain$ represent the current heaviest EC chain according to the participant (that extends $baseChain$ by design)
Let $e'$ be the epoch at which the participant is starting the new f3 instance (following decision on #173 , $e' >= baseChain.head().epoch + 2$).

Then:

If ECChain.head().epoch < e', participant sets its proposal to ECChain
If ECChain.head().epoch = e', participant sets its proposal to ECChain[-1] (remove the head tipset, from the current epoch e')
ECChain.head().epoch cannot be > e' by design (e' is the current epoch)

I think this is done: it is specified in the FIP so the decision is make. @Kubuxu do you need this issue open to track some implentation task?

We don't have the head offset currently. Looking at the FIP pseudocode, it appears "delete the head always", as I'm assuming CurrentEpoch() is the epoch of the chain head and collecting the chain for HeaviestUnfinalizedChain will return that head.

It's difficult to be precise with language here. I had interpreted the FIP's use of CurrentEpoch() to mean the one for which new blocks are currently being transmitted but a tipset could not yet be confirmed by a subsequent block pointing to it. This is why it is always deleted.
What exactly do you mean by the epoch of the chain head? Is it the epoch of the blocks currently being proposed? Or the epoch of the tipset to which the most recent possible block points?

What you wrote here makes sense.