Importing Berlin

I've used ./importer --oneshot=/ with the binary version and this (extracted) OSM file: http://download.geofabrik.de/europe/germany/berlin-latest.osm.bz2
The importer crashes with Unknown turn restriction no_entry:
- Running convert_osm on /Users/jerry/Downloads/berlin-latest.osm
Read /Users/jerry/Downloads/berlin-latest.osm (1,040)... 162.1816s
OSM doc has 5522394 nodes, 818090 ways, 14293 relations
processing OSM nodes (5,522,394)... 4.4821s
processing OSM ways (818,090)... 6.3400s
Relation 1653527 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 3455436 has unhandled member role inner, ignoring it
Relation 2107105 has unhandled member role inner, ignoring it
thread 'main' panicked at 'Unknown turn restriction no_entry', map_model/src/raw.rs:373:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
dropping Timer while doing progress processing OSM relations, due to panic?

The OSM importing code is extremely strict about stuff I haven't seen before, to prevent regressions when converting regularly used maps. This one is easy to fix (edit map_model/src/raw.rs:373) if you've built from source. You're bound to hit a few more issues though. I'll download the Berlin map and fix whatever I encounter.

The entire Berlin area is huge. Could we start with a smaller area? Not neighborhood small, maybe the general city center area, just not 1.1GB. :)  I use geojson.io or geoman.io to draw polygon boundaries.

Wow, thanks for doing this.
Sorry. I didn't know how much this file covered. Of course.
How about this for the core city centre:
"coordinates": [
          [
            [
              13.361434936523436,
              52.49532344352079
            ],
            [
              13.432674407958984,
              52.49532344352079
            ],
            [
              13.432674407958984,
              52.53199183474197
            ],
            [
              13.361434936523436,
              52.53199183474197
            ],
            [
              13.361434936523436,
              52.49532344352079
            ]
          ]
        ]

and this for a larger part of the city which is surrounded by the circle train:
"coordinates": [
          [
            [
              13.271141052246094,
              52.45935663683681
            ],
            [
              13.47644805908203,
              52.45935663683681
            ],
            [
              13.47644805908203,
              52.561116680853836
            ],
            [
              13.271141052246094,
              52.561116680853836
            ],
            [
              13.271141052246094,
              52.45935663683681
            ]
          ]
        ]

I can try to create those filter files.

Filter files: berlin-polys.zip
I've tried another time with berlin-centre.poly but am still getting the same error.

Thanks! I got the large one working. You can put this file in data/system/maps: https://www.dropbox.com/s/4uia1pk9aj1d8gd/berlin.bin?dl=0
My GPU half-melted during loading and my mouse was briefly unresponsive, but in the end, only took 20s to load. :)
I'll commit a fix in a moment that deals with almost all of the issues. There are a few intersections where seemingly different combinations of roads wind up with exactly the same turn geometry, which breaks some assertions. I want to figure out why that's happening instead of relaxing that code permanently.

Interesting: I think Berlin is the first place I've run across with OSM parking tags filled out somewhat. It's been quite painful mapping them in Seattle. Odd that the coverage in Berlin is so spotty; IIRC from a prior visit, there's definitely more than just these areas

Wow! That's amazing. Thanks a lot for your quick help.
Unfortunately, abstreet is crashing when trying to load the Berlin map. output.txt contains:
Loading map ../data/system/maps/berlin.bin

Reading ../data/system/maps/berlin.bin: 0/149 MB... 0.0005s

../data/system/maps/berlin.bin is missing or corrupt. Check https://github.com/dabreegster/abstreet/blob/master/docs/dev.md and file an issue if you have trouble.

invalid value: integer `512`, expected variant index 0 <= i < 8

Interesting: I think Berlin is the first place I've run across with OSM parking tags filled out somewhat.

That's great!

I found what I think is a problem with Bayernring in OSM:

https://www.openstreetmap.org/way/722013168
https://www.openstreetmap.org/way/722013169
The first one is an overlapping road of the second. I think the second one needs to be split at the point where the road attributes are different. Are you part of the OSM community in Berlin? If not, I can attempt a fix or raise the issue on the appropriate slack.

The crash is because the binary map format has changed since Sunday. That's the price of rapid iteration. :(  If you can build the game crate from source, it'll handle the new format. If not, https://github.com/dabreegster/abstreet/suites/866601914/artifacts/10031908 is a Mac binary from last night that should work

The new Mac binary is working with the Berlin map. Fantastic! Thanks so much.

Are you part of the OSM community in Berlin?

No sorry, I'm not. But I'm part of CityLAB Berlin. I should introduce myself properly in the next few days. And maybe my colleagues can answer some of your questions on the Berlin data.
Thanks so much! 🙏 That was the quickest support.

About the parking data, my colleague @Lisa-Stubert mentioned an official data set from 2014 that contains this data in some format. It might be useful to fill in the gaps on OSM.
FIS-Broker > Straßenbefahrung > Parkfläche > WFS address

The data set contains polygons. @Lisa-Stubert mentioned we might need polygons instead.

The direct link to WFS address seemed broken, and I couldn't find Parkfläche on the first page. Is there a different direct link?
Seattle has a perhaps similar dataset, blockface. I'm matching that to the nearest side of each road here. We could do something similar for this dataset.
Ideally the data winds up in OSM, so anybody can use it, and it can be maintained more easily. OSM has strict rules around importing data automatically. Regardless of an import, theres a tool in A/B Street to update parking tags in OSM a little more conveniently than other editors.

The direct link to WFS address seemed broken, and I couldn't find Parkfläche on the first page. Is there a different direct link?

I'm not familiar with the workflow yet, but a former colleague has written this guide on how to use that data source: https://lab.technologiestiftung-berlin.de/projects/fisbroker-to-qgis/en/
Apparently you can also load it like this (with GDAL installed): ogr2ogr -f gpkg Parkflächen.gpkg WFS:"https://fbinter.stadt-berlin.de/fb/wfs/data/senstadt/s_Parkflaeche"
I'm going to get back to this next week.

https://fbinter.stadt-berlin.de/fb/index.jsp?loginkey=zoomStart&mapId=k_StraDa@senstadt&bbox=383640,5815972,386377,5817593 this is the direct link to the data record of the "road traffic". It contains all possible areas and objects of road traffic for Berlin. The dark grey areas are the parking areas. They can only be downloaded via the WFS service @jvolker posted.

I've just tried to load the berlin map in A/B street compiled from source on Windows and received this error:
switch map...
Wrote ../data/player/camera_state/montlake.json
load map...
Loading map ../data/system/maps/berlin.bin
Reading ../data/system/maps/berlin.bin: 0/149 MB... 0.0001smemory allocation of<PHONE_NUMBER>3840414750 bytes failederror: process didn't exit successfully: `C:\Users\PC\Desktop\AB-Street\abstreet\target\release\game.exe` (exit code: 0xc0000409, STATUS_STACK_BUFFER_OVERRUN)

Do you have any idea what could cause this? Thanks.

The binary format of the map changed yesterday.
I can regenerate the Berlin map and give you a new dropbox link in the next ~30 mins

https://www.dropbox.com/s/2l2muvha0dubuvp/berlin.bin?dl=0
The simulation crashes 3m58s in. I can investigate soon; I suspect this is the same as https://github.com/dabreegster/abstreet/issues/143#issuecomment-654488564 -- there are some bus stops mis-tagged in OSM. I'll improve the error message to find these better, or filter out the invalid ones.

 Not high priority to figure out what's wrong here.

Thanks. It's working again.

Just checking, are you blocked on me for anything besides importing and matching parking data? (which I'm not likely to get to anytime super soon)

Thanks for asking @dabreegster. It seems the most important thing is to get trip data to get a more realistic simulation. We are currently looking into data sources and are going to post them here.

Hey Dustin!
So we've done some brainstorming and come up with some additional data sources that might be useful for the project. Maybe you can take a look and give us an indication which data sources you see as more or less useful, or what's still missing?
The biggest thing we don't have right now is any sort of traffic pattern model like the SoundCast model. It's unclear to me to what extent this data exists in Berlin at all, and to the extent it does, it's not available as open data and we currently lack the contacts to rustle up that data.
So what do we have?

Public transit data (GTFS): https://daten.berlin.de/datensaetze/vbb-fahrplandaten-gtfs (available as a regularly-updated upload; there is also an API, but it requires first contacting the local transit operator to get access)
Car traffic speed data: https://movement.uber.com/explore/berlin/speeds/query?dt[tpb]=ALL_DAY&dt[wd;]=1,2,3,4,5,6,7&dt[dr][sd]=2020-03-01&dt[dr][ed]=2020-03-31&ff=&lang=en-US (this is a dataset made available by Uber; shows average speeds over various road segments in both directions – could be useful as a proxy for traffic flows?)
Traffic volume data: https://fbinter.stadt-berlin.de/fb/index.jsp?loginkey=zoomStart&mapId=wmsk_07_01verkmeng2014@senstadt&bbox=388185,5818529,395105,5822530 (This is a very general dataset, available via WFS, showing estimated number of vehicles driving over major streets in a 24 hour period – this data is based on traffic counts originally gathered in 2014, so it's pretty outdated)

What might we have?
I looked at the dataset containing information on "blockfaces" to see what it contained (wasn't familiar with this term myself). Seems like it includes data on things like zones (e.g. loading and unloading zones, no parking zones, etc.) as well as metered parking/parking fees. Not sure what out of the blockfaces dataset is most relevant for A/B Street – if it's just about knowing where there is streetside parking, then the aforementioned "Straßenbefahrung" dataset is probably our best bet there. If you also want information on no-parking zones etc., I think we can possibly also get those out of the Straßenbefahrung dataset, but I'm not certain. We definitely won't be to get get good data on where parking is metered (and under what circumstances); there isn't currently a comprehensive city-wide dataset for that.
We may be able to rustle up some data on off-street parking – @Lisa-Stubert wanted to check out another large geospatial dataset we have to see if parking lots and parking garages are in it (although if they are, my guess is we won't have data on capacities, but maybe I'll be proven wrong).
Let me know what questions you have. If there's an acute need for data that has not yet been addressed, also let me know about that and we can go sleuthing...

Thanks for doing this research!

The GTFS upload will be useful eventually. The transit modelling needs lots of work first; currently just a single bus is created per route, and it runs all day. No need for API access; the schedule information is used when building the map as a one-time process.
Speed and traffic volume could be helpful. I've seen research papers before talking about how to use volume data to infer a set of start/end points that would cause the measured volume counts. I have no familiarity with the algorithms for doing this; it'd be future work to implement them or find an existing tool that does it.
Straßenbefahrung: It sounds like this is the equivalent of the Seattle blockface data indeed. We could use it just to infer the existence of space along the shoulder of the road for street parking. The data in Seattle wasn't adequate; it says "no parking restrictions" on narrow roads that physically had no space for parking. So I use it as a first pass, manually look at areas of the map with odd road geometry, then fix upstream in OpenStreetMap using a tool.
There's no modelling of metered parking, resident-only restrictions, 2 hours max, etc yet. No immediate plans to figure this out either.
Offstreet parking with garages/lots could be helpful. We can estimate capacity from the geometry. Some lots are already in OSM: https://www.openstreetmap.org/way/669540789

I think the biggest thing that would help generate better trip data would be census.  Then there's the American Commuter Survey that has questions like "how many cars owned per household" and "how many trips > 5 miles taken per day." The sample size is probably really small, but it's better than nothing, and I think the survey results are correlated with census tracts. So from this, the really simple trip generation model would take the number of people inside a tract, randomly distribute them to houses in that area (using OSM's residential vs commercial building tagging), guess a workplace/school/common destination for them using stats on how far people in that area travel, and so on.
I see https://en.wikipedia.org/wiki/Demographics_of_Berlin, and drilling down, https://en.wikipedia.org/wiki/Tiergarten,_Berlin is a small area with a population of ~12.5k. So I think we could use this! I haven't looked for where the wiki article pulls data; if there's a more consolidated format that ideally has the GPS coordinates of each neighborhood with this population count, that'd be a huge step forward.

With respect to the Census Tracts / finding a corollary in Berlin:
So there are various administrative units that Berlin as a city is broken up into. A coworker actually did a nice breakdown of them (in English) here: https://lab.technologiestiftung-berlin.de/projects/spatial-units/en/. The example you found, Tiergarten, is an example of an Ortsteil/locality. Of the top of my head, I'm not actually sure what the smallest possible unit is (there are a few different systems for breaking up the city that are used in different concepts, so it's not necessarily straightforward to arrange all of the units from smallest to biggest. I guess what we want is the smallest possible unit for which we also have demographic data?
I think the smallest unit for which we would have overall population data is a Planungsraum/Planning Area. The population data is available here: https://daten.berlin.de/datensaetze/einwohnerinnen-und-einwohner-berlin-lor-planungsräumen-am-31122018
We also have the geometry of the Planungsräume available in various geospatial formats: https://data.technologiestiftung-berlin.de/dataset/lor_planungsgraeume/en (I think this file is slightly outdated though, we'll have to update it first).
I don't know if we'll be able to find additional census-like data for that level of specificity, however. Within Berlin itself I haven't seen any sort of survey data around mobility habits.
@Lisa-Stubert found this dataset, which probably has what we want? https://daten.clearingstelle-verkehr.de/279/ It includes a regional level, where an area is broken up into a grid of squares that are at least 500x500m and have at least 500 residents in them. The data, as far as I can tell, would include information on the average trip length from a given cell, whether the people living there own cars, etc. Unfortunately, it is not available as open data and I suspect it is only available for a fee. Moreover, the data seems like it is likely to be fairly complex, so even if we were to acquire it, I think it might be beyond our resources at least to integrate it into the program.
I can keep poking around to see if I find anything else at the Berlin level, but at least right now, we definitely have population data for sub-units of Berlin.

The spatial breakdown article is quite helpful! Thanks for finding the datasets. I think these will work for figuring out how many residents to assign to buildings. Would you mind sanity checking my interpretation of the population CSV -- Google Translate seems fine, but you never know. I found Körnerstraße in the KML, with spatial_name of 01011104. That  matches RAUMID in the population csv. The E_E column is total residents, so that's 4759 in that area, right?
The 500x500m trip length dataset may be helpful, but if it's not open, not a priority.
Here's a proposal for some next steps:

I'll add Berlin to the set of maps I regularly convert when the map format changes. The larger boundary suggested here turned out a bit too big, so I'll start with the city core. If anyone wants to use geojson.io or geoman.io to draw multiple boundaries, just send the coordinates along. The regions can overlap slightly, they don't have to cover everything (so not a strict partition in the mathematical set sense) -- an example is:

I'll make the import pipeline grab the shapefile and population CSV, then emit a file that'll let us visualize the regions with the population count.

I'll start researching approaches to reasonably distribute the number of people in a region into individual buildings. @matkoniecz came up with some heuristics for guessing building sizes using OSM tags.

First two steps done. The importer grabs the shapefile and the population CSV, then adds a num_residents attribute to the shapefile. Here's a way to quickly visualize it:

I've started including Berlin in the default binary release, because the city center is tiny (only adds 20MB). This'll hopefully make it easier to iterate without any of you needing to build from source or run the updater. I'll post the build here when it's done; github is having partial outages right now.

Linux: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329941
Mac: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329942
Windows: https://github.com/dabreegster/abstreet/suites/924669830/artifacts/11322491
To follow the video above, you'll also need https://www.dropbox.com/s/pbovhfr6zw0s0ob/planning_areas.bin.zip?dl=0. Unzip so that data/input/berlin/planning_areas.bin exists

The spatial breakdown article is quite helpful! Thanks for finding the datasets. I think these will work for figuring out how many residents to assign to buildings. Would you mind sanity checking my interpretation of the population CSV -- Google Translate seems fine, but you never know. I found Körnerstraße in the KML, with spatial_name of 01011104. That matches RAUMID in the population csv. The E_E column is total residents, so that's 4759 in that area, right?

Yep, this is the correct interpretation!

Thanks a lot, @dabreegster.

Linux: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329941
Mac: https://github.com/dabreegster/abstreet/suites/925031782/artifacts/11329942
Windows: https://github.com/dabreegster/abstreet/suites/924669830/artifacts/11322491

Those versions have a "trips between home and work" traffic setting. Is this already using the population count?

Two quick observations:

I've noticed using that setting the area around "Kottbusser Tor" is a lot busier in reality.
The CPU is at 100% and the simulation very slow using that setting. I assume the city center area is still too big.

I've noticed using that setting the area around "Kottbusser Tor" is a lot busier in reality.

Are there also unusually quiet areas? (for now Kraków has some weird hot spots, especially pedestrian hot spots and some unusually quiet areas, mostly as indirect results of issues that I already reported)

The CPU is at 100% and the simulation very slow using that setting. I assume the city center area is still too big.

For me it is also consuming plenty of resources with laggy simulation. Not sure whatever it is simply result of many trips, or is there something that triggers unusually poor performance.

Those versions have a "trips between home and work" traffic setting. Is this already using the population count?

Not yet, it was added a little later. I'll regenerate the build and post new links soon, unless you get the Rust compiler updated successfully first.

I've noticed using that setting the area around "Kottbusser Tor" is a lot busier in reality.

I'm seeing quite a few possibly unusual patterns. In the morning rush hour, there are 20,000 pedestrian trips and climbing, but under 1000 drivers. This is the logic used to pick driving/walking/biking/transit. If you had to guess a trip mode in Berlin based on any factors you can imagine, what kind of heuristics would make sense?
Some of the weird issues include cars backing up at Torstraße because of pedestrians getting the right-of-way at crosswalks here. Likely a bug with the stop sign policy, and also exasperated by A/B Street inferring crosswalks at all possible crossings and ignoring data in OSM right now.

There's also a massive number of residents matched to a single building in the SW corner. From the population layer:

I think it's because the Nollendorfplatz area has 16,000 residents, but the polygon is mostly out-of-bounds. Maybe it'd be better to just linearly scale the number of residents based on the percent area of the polygon overlapping with the map boundary.

The CPU is at 100% and the simulation very slow using that setting. I assume the city center area is still too big.

It's more that the number of residents is huge -- 360,000 -- and the trip generation tries to make everybody experience a morning and afternoon rush. There are a number of simulation issues also causing problems, like the stop sign / crosswalk problem I mentioned. And I have never run with so many pedestrians at once; I'll look into profiling and optimizing code dealing with them.

in the morning rush hour, there are 20,000 pedestrian trips and climbing, but under 1000 drivers. This is the logic used to pick driving/walking/biking/transit.

I noticed the same in Kraków, I attributed it to transit/drivers trip spawns failing for some reason, what seems supported by logs. That is why I keep reporting issues blocking bus/car/bicycle travel. In addition in Kraków many car trips fail because there was no space to park the car (during mapgen), maybe the same happens here.

unless you get the Rust compiler updated successfully first.

Got the game compiled but the city center map is gone. How can I reimport it? import.sh --map berlin seems to quit unsuccessfully.

Are there also unusually quiet areas?

The area I've mentioned is unusually quiet in A/B street compared to what I've seen in reality. Could it be because it's relatively close to the border? How are the work targets generated?

Some of the weird issues include cars backing up at Torstraße because of pedestrians getting the right-of-way at crosswalks here. Likely a bug with the stop sign policy

That particular crossing does have traffic lights. Stop signs are only relevant in case the traffic lights are out of order.

Got the game compiled but the city center map is gone. How can I reimport it? import.sh --map berlin seems to quit unsuccessfully.

Try ./import.sh --raw --map --city=berlin. Confusingly, the city is always assumed to be seattle, unless you override it. I'll make the tool smarter and realize that berlin_center implies a city.

Could it be because it's relatively close to the border? How are the work targets generated?

Possibly the number of residents matched to the area is low or there aren't enough workplaces around. (layer > amenities will show you those.) I can dig in later today.

That particular crossing does have traffic lights. Stop signs are only relevant in case the traffic lights are out of order.

https://www.openstreetmap.org/node/3792544915
Ah, there are two styles for tagging traffic signals. This is using the more detailed representation showing where the lights are for each incoming road. I can add support for this soon.

This is the logic used to pick driving/walking/biking/transit.

Trying to understand the logic:

What does "transit" mean in that case? Public transport?
What distance applies to the following?

if rng.gen_bool(0.005) {
        // low chance for really, really dedicated cyclists
        return TripMode::Bike;
    }
    if rng.gen_bool(0.3) {
        // try transit if available, will
        // degrade into walk if not available
        return TripMode::Transit;
    }

We'll see what data we have for it. Otherwise we could make a guess.
If I understand correctly, currently:

there are no car or public transport trips for distances shorter than 3miles/4.8km? I think there would be.
0.5% would cycle if the trip was further than 3miles/4.8km? I think it would be more.

"Transit" is public transport, yes.
The distance is measured by the walking route. Sometimes the distance could be much longer or shorter by driving, but usually this is an OK proxy.
Your understanding about the rest seems correct. This is a function we've kind of made up out of vague intuitions, and tuned for Kraków. Mode choice is a complicated topic I don't have an understanding of yet. If you have alternative ideas or come across any academic papers that are easy to understand, implementation is the easy part.

Try ./import.sh --raw --map --city=berlin.

I think, it's still the same unfortunately. Another windows opens, displays some console output and closes again. In A/B street Berlin is still not visible.

You're running from the git repo, not the binary release, right? I don't understand why a second terminal is opening; are you typing that command into a first mac terminal? You could add > log.txt at the end to redirect output to a file, if that helps.

You're running from the git repo, not the binary release, right?

Yep.

are you typing that command into a first mac terminal?

No, it's a PC running Windows.

You could add > log.txt at the end to redirect output to a file, if that helps.

The file is empty, unfortunately.

Maybe 2> log.txt (stdeer capture) will work?

Maybe 2> log.txt (stdeer capture) will work?

No, nothing, unfortunately.

This is a function we've kind of made up out of vague intuitions, and tuned for Kraków

"tuned" is overstatement, it was more of "AB Street is no longer crashing and there are some pedestrians, cars and cyclists" and "problem of bicyclist traffic jams, with bicycle traffic over 100 times reality is reduced to 5 times reality".
Feel free to change numbers or tweak it based solely on Berlin, do not worry about Krakow (though if you want you may test it also there).

Ah, maybe running a separate shell script on Windows launches a new terminal. Try:
RUST_BACKTRACE=1 cargo run --release --manifest-path importer/Cargo.toml -- --raw --map --city=berlin
I'm not sure about Windows syntax, but you could also try . import.sh --raw --map --city=berlin. The leading .  means run all of the commands in a file, but ignore the first line that says what shell to execute with.
Also I noticed you're doing --city=berlin_center; it should be --city=berlin. The difference isn't interesting when a city only has one map, but we'll probably split into multiple maps at some point, like Seattle. The city name is berlin and the only map there is called berlin_center, currently.

Thanks, the following has worked:
set RUST_BACKTRACE=1 && cargo run --release --manifest-path importer/Cargo.toml -- --raw --map --city=berlin
But I'm now getting the following errors:
   Compiling importer v0.1.0 (C:\Users\PC\Desktop\AB-Street\abstreet\importer)
warning: unreachable expression
  --> importer\src\main.rs:96:13
   |
91 | /             panic!(
92 | |                 "Can't do --scenario or --scenario_everyone without the scenarios feature \
93 | |                  compiled in"
94 | |             );
   | |______________- any code following this expression is unreachable
95 |               // Nonsense to make the type-checker work
96 |               (Some(true), Some(true))
   |               ^^^^^^^^^^^^^^^^^^^^^^^^ unreachable expression
   |
   = note: `#[warn(unreachable_code)]` on by default

warning: unused variable: `maybe_popdat`
  --> importer\src\main.rs:80:10
   |
80 |     let (maybe_popdat, maybe_huge_map) = if job.scenario || job.scenario_everyone {
   |          ^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_popdat`
   |
   = note: `#[warn(unused_variables)]` on by default

warning: unused variable: `maybe_huge_map`
  --> importer\src\main.rs:80:24
   |
80 |     let (maybe_popdat, maybe_huge_map) = if job.scenario || job.scenario_everyone {
   |                        ^^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_huge_map`
warning: unused variable: `maybe_map`
   --> importer\src\main.rs:112:13
    |
112 |         let mut maybe_map = if job.raw_to_map {
    |             ^^^^^^^^^^^^^ help: if this is intentional, prefix it with an underscore: `_maybe_map`

warning: variable does not need to be mutable
   --> importer\src\main.rs:112:13
    |
112 |         let mut maybe_map = if job.raw_to_map {
    |             ----^^^^^^^^^
    |             |
    |             help: remove this `mut`
    |
    = note: `#[warn(unused_mut)]` on by default

warning: function is never used: `adjust_private_parking`
   --> importer\src\seattle.rs:116:8
    |
116 | pub fn adjust_private_parking(map: &mut Map, scenario: &Scenario) {
    |        ^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: `#[warn(dead_code)]` on by default

warning: 6 warnings emitted

    Finished release [optimized] target(s) in 6m 27s
     Running `target\release\importer.exe --raw --map --city=berlin`
- Working on all berlin maps
import map data...
- Missing data/input/berlin/osm/berlin-latest.osm.pbf, so downloading http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf
- Running "curl" "--fail" "-L" "-o" "tmp_output" "http://download.geofabrik.de/europe/germany/berlin-latest.osm.pbf"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 57.2M  100 57.2M    0     0  11.4M      0  0:00:05  0:00:05 --:--:-- 11.1M
- Running "mv" "tmp_output" "data/input/berlin/osm/berlin-latest.osm.pbf"
thread 'main' panicked at 'Failed to run "mv" "tmp_output" "data/input/berlin/osm/berlin-latest.osm.pbf": Os { code: 2, kind: NotFound, message: "Das System kann die angegebene Datei nicht finden." }', importer\src\utils.rs:132:13
stack backtrace:
   0: backtrace::backtrace::trace_unsynchronized
             at C:\Users\VssAdministrator\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-0.3.46\src\backtrace\mod.rs:66
   1: std::sys_common::backtrace::_print_fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:78
   2: std::sys_common::backtrace::_print::{{impl}}::fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:59
   3: core::fmt::write
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libcore\fmt\mod.rs:1076
   4: std::io::Write::write_fmt<std::sys::windows::stdio::Stderr>
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\io\mod.rs:1537
   5: std::sys_common::backtrace::_print
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:62
   6: std::sys_common::backtrace::print
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\sys_common\backtrace.rs:49
   7: std::panicking::default_hook::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:198
   8: std::panicking::default_hook
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:218
   9: std::panicking::rust_panic_with_hook
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:486
  10: std::panicking::begin_panic_handler
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:388
  11: std::panicking::begin_panic_fmt
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:342
  12: importer::utils::osmconvert
  13: importer::utils::download
  14: importer::berlin::osm_to_raw
  15: alloc::collections::btree::search::search_tree
  16: csv::reader::Reader<R>::deserialize
  17: std::rt::lang_start_internal::{{closure}}
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\rt.rs:52
  18: std::panicking::try::do_call
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:297
  19: std::panicking::try
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panicking.rs:274
  20: std::panic::catch_unwind
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\panic.rs:394
  21: std::rt::lang_start_internal
             at /rustc/5c1f21c3b82297671ad3ae1e8c942d2ca92e84f2\/src\libstd\rt.rs:51
  22: main
  23: invoke_main
             at d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:78
  24: __scrt_common_main_seh
             at d:\A01\_work\6\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288
  25: BaseThreadInitThunk
  26: RtlUserThreadStart
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
import map data took 5.4766s

- import map data took 5.4766s

- import map data took 5.4766s
error: process didn't exit successfully: `target\release\importer.exe --raw --map --city=berlin` (exit code: 101)

You can ignore the warnings. It tried to execute this command: mv tmp_output data/input/berlin/osm/berlin-latest.osm.pbf. Looks like Windows calls it move, not mv.
I'll switch importer to use system calls instead of relying on Linux/Mac commands when possible. Not going to get to that for a few hours. As a temporary workaround, you could edit importer/src/utils.rs, look for all Command::new, and if you get an error about one of them, change it appropriately. I think mv becomes move, cp -> copy, and rm -> del (possibly messing with the -rfv flags).

Number of people in houses around here down from 360k to 180k, after adjusting for the planning areas that only partly overlapped the map. Does this seem generally more right to you?

A few additional data sources to throw in there:

I found this study published in 2018 that I think actually gets at this census-level data you mentioned before, @dabreegster. Basically one of the universities here did a survey of 40,000 Berliners in which they were asked about their mobility habits (e.g. modal split, types of trips, etc.). They have detailed reports for each of Berlin's 12 districts/Bezirke. However, there is currently no open data available, just PDFs. We could try to reach out to this university however and see if they would be interested in sharing their data, perhaps they would find our project interesting: https://www.berlin.de/sen/uvk/verkehr/verkehrsdaten/zahlen-und-fakten/mobilitaet-in-staedten-srv-2018/

For bikes, we do have open data from the automatic counting stations the city has. The table is available here: https://www.berlin.de/sen/uvk/verkehr/verkehrsplanung/radverkehr/weitere-radinfrastruktur/zaehlstellen-und-fahrradbarometer/#dauer (if you scroll there is an Excel file). It shows the hourly counts for each station over several years. Using that, we could at least get pretty accurate counts of how many bikers are going past a given point in the city on a given day / at a given time (there are 17 stations across the city). We did an analysis with the data up to 2017 a couple years ago, so we do have some already have some cleaned up versions of this data that theoretically could be used as a basis? Not sure what data specifically would be most helpful to have here though (e.g.: should the hourly data be aggregated to a certain level for each location?)

The travel surveys look really promising! Right now, our approach knows roughly how many people reside in each building, but we don't know how far they travel, or how. If we could put together a table of (origin district, destination district, percentage of bike trips, pct of driving, ...), then we could pretty easily do something more realistic. I can't understand the PDFs too easily, but if they have this quadratic table of trips between every pair of districts, 144 things isn't unreasonable to manually copy from a PDF.
Bike counts may be helpful, but I don't know how to turn the data into individual trips. This is a sample paper describing how to do it. It's way beyond my current ability to understand.
The more I read about https://activitysim.github.io/activitysim/index.html, the more I'm convinced this could be the more trustworthy way forward. Seattle's Soundcast model makes use of this. Glancing through the amount of detail in the different parts of their model, there's no hope of reproducing this level of rigor. I think A/B Street ought to require trip data as input. We can do extremely simple approximations that just look at OpenStreetMap data and maybe some extra census counts, but I don't know if it's worth trying to recreate what ActivitySim has done.

An update: there's renewed interest in getting a good travel demand model for Berlin, from the folks creating CargoBikeCity. Since July, a few more ideas/options for generating demand data have arisen:

@behrisch clued us in to https://github.com/matsim-scenarios/matsim-berlin
https://github.com/DLR-VF/TAPAS was released
https://github.com/lcodeca/SUMOActivityGen
We renewed our efforts to generate a simple model using only OSM data and population counts: #424

@dabreegster great to see this long thread about Berlin. We have a dedicated OSM UserGroup about "Traffic evolution" in Berlin. More at https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende. Some things we work on:

We are looking into extending the tagging for bike lanes with new, safety related tagging and where required separate ways to represent bike-lane-only turns (around trees and such) – examples at https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Radwege#Tagging-Beispiele. This is still experimentell, however.
Same for pedestrian routing; this experimental area has full separately mapped sidewalks https://www.openstreetmap.org/node/4794919616#map=17/52.47639/13.42334
We are looking at improving the pedestrian crossing mapping and tagging – this is in an early stage https://wiki.openstreetmap.org/wiki/Berlin/Verkehrswende/Fußwege#Tagging-Beispiele
We mapped all Neukölln with the parking:lane Schema. This allows reports like https://www.openstreetmap.org/user/Supaplex030/diary/393423.

I would love to see the map with lanes and such running to start checking the OSM data and adjust it where needed. What is your recommendation how to get this running locally (Mac) in this UseCase?

What is your recommendation how to get this running locally (Mac) in this UseCase?

Where you are stuck? At running ABStreet at all or at running specifically Berlin?

Hi @tordans! I'm constantly amazed by the level of detail in OSM being added, and the new proposals on the Verkehrswende wiki page look exciting. I'm quite interested in improving the accuracy of the map model in A/B Street from this data, but some of it is going to be quite difficult.
One big problem is that A/B Street doesn't yet handle sidewalks and cyclepaths mapped separately from the "main" road way. #330 and #161 have details. In short, it's hard for a few reasons:

The simulation needs to relate sidewalks with the rest of the road to handle "modality transfers", like walking from a building to the road, then starting to bike or entering a parked car. When the sidewalk isn't explicitly associated with the rest of the road, inferring these connections is hard.

Rendering and inferring good geometry for separate ways is hard. Part of the reason is due to widths -- since I haven't seen width explicitly tagged in most places I've looked, all lanes have fixed widths. Separate paths tend to be very close to the road, and because the inferred width is wrong or the GPS is slightly out of alignment, the generated roads tend to overlap.

The traffic simulation has trouble when there are lots of short "roads" broken up by extra "intersections." Separate crosswalks aggravate this problem even more.

That being said, you could enable using more of the OSM sidewalk data by using config like this.
I'd love to work on fixing these problems, but I tend to over-commit how much I can work on. If anybody from your group is interested in helping with development, that would be ideal. I can mentor if Rust isn't a familiar language, but some programming experience is necessary to make progress.
One good first step could be to extend the area of Berlin in A/B Street. We just have the one shape right now. If Neukölln is a useful area to focus on, for example, then we could chop up the area into more pieces. This is best done by locals familiar with the city to choose meaningful boundaries. https://dabreegster.github.io/abstreet/howto/new_city.html#including-the-city-to-ab-street-more-permanently has more info -- geojson.io is a useful tool for drawing the polygons.
I spotted this from y'all's Telegram group:

Once we have a polygon covering this area and are regularly importing, I can look into specific problems like this.

What is your recommendation how to get this running locally (Mac) in this UseCase?

https://dabreegster.github.io/abstreet/dev/index.html has instructions for compiling locally -- is that what you meant?

Hey Dustin,
I am also happy to have discovered this thread (I'm another member of the "Verkehrswende" group). Neukölln could be an perfect playground for testing detailed street information. As you've already noticed, we have parking lanes mapped all over the place here. In addition, we are currently starting to systematically collect other data that could be interesting for traffic modelling: For example, road widths (a discussion for this purpose resulted in the OSM tagging "width:carriageway" a few months ago, see OSM-Wiki). Most roads here have no markings, so width data is most relevant for estimating traffic flow. Do you already take width data into account in any way?
As mentioned by @tordans we have also started to record separate sidewalks. "sidewalk=separate" on a highway is an indicator for this, which you can evaluate even when analysing the street lines without having to generate relationships to separate geometries. We need to improve this a bit, because an indication like "sidewalk:both=separate" or "sidewalk:left=separate" would be clear to show on which sides of the road there is a sidewalk. Detailed attributes for the sidewalks (like width – if mapped at all) are lost with separate mapping, but not the information that there is a sidewalk at all. We do it similarly for cycle tracks.
There are other details like curbs, curb extensions or certain forms of crossings that we are working on, but the two points mentioned above (street width and the street vs. pavement problem) are probably the most interesting for you.
I'd be happy if our well-mapped neighbourhood is enriching your developments and perhaps encourages other mappers to map in more detail. Applications like A/B Street and the analyses that are possible with them show what potential can be found in open data! I'm not a programmer, unfortunately, so I can't help with A/B Street (but I am happy to contribute on local or tagging issues).
P.S. If you're interested: I'm currently working on a parking analysis (with QGIS) for this Neukölln district, which I'll finish during the next weeks: see some first impressions here.

sidewalk=separate" on a highway is an indicator for this, which you can evaluate even when analysing the street lines without having to generate relationships to separate geometries.

This is implemented already, though maybe it can be tweaked a bit. AB is smart enough to handle it specially on major oneway roads (dual carriageways).

Do you already take width data into account in any way?

Not really, except for narrow=yes. https://github.com/dabreegster/abstreet/blob/master/map_model/src/make/initial/lane_specs.rs is the one place that transforms OSM tags into A/B Street's lane model. Maybe more interesting than the (disorganized) code are the tests at the bottom. You've probably noticed that regular driving lanes and bike lanes both have the same width in A/B Street. I would very much like to have more reasonable defaults, and incorporate the exact lane and road widths if they're tagged. But it might make editing lanes in the game, one of the main actions the player can take, more complicated. Do we have to figure out if the road has enough width to support a lane type transformation? Do we need to shift the geometry of the lane slightly? Does the UI for changing lanes need to get more detailed -- maybe closer to streetmix.net? This is the sort of thing I'd love to tackle when I get time or find somebody interested in designing+implementing this.

we have also started to record separate sidewalks

Things like sidewalk:left=separate would be a massive improvement over sidewalk=separate, in my opinion, because it would let me defer the complex problem of handling separate sidewalks at least for a little while. Although extremely unwieldy to work with, a relation would be even better -- something with 2 ways as members, for example -- role="main_road", role="sidewalk". I can imagine some algorithms to effectively infer this relation automatically by "tracing" the separated sidewalks and figuring out where they have a crossing over the road, but also there are lots of edge cases.
I still think the best next step is to get more boundary polygons for areas around Berlin in.

@SupaplexOSM, I just found your micromap thanks to @tordans in the OSMUS Slack. This is frankly the coolest thing I have seen in a long time. It's an unprecedented level of detail, yet immediately clear how to read -- the design and color choices are awesome.
I'm still reading through the blog posts and code, so apologies if some of these questions are answered there.

Do you have a method for calculating the width of a road from the mapped curbs? Does QGIS have a tool like this that can be applied automatically?
When you know the total width of a road, do you have heuristics for dividing up that width amongst some number of lanes with different purposes?

Since last December, A/B Street has made progress representing roads and lanes with modifiable width: https://a-b-street.github.io/docs/project/retrospective/index.html#road-editor
We've also made some progress logically and geometrically consolidating complex junctions. I just published a long article about how this works, https://a-b-street.github.io/docs/tech/map/geometry/index.html.
Finally, I'm curious how your parking space analysis is being received by the community. Are you using this to argue that there's too much public space being used to store vehicles? Or is this project more aimed at representing data in OSM instead of other sources?

Oh, I'm glad you like the map! Unfortunately I'm not using such methods and I don't know if they would be possible... I am actually rather a non-expert in scripting etc. (and at the beginning of the project also in GIS) and therefore often have to look for simple solutions... There are two elements in the map so far that are not 100% generated from OSM:

One are the carriageway areas, which I have as separate shapes. They are a "hand-made" mixture of OpenData from the city of Berlin with (partly outdated) kerb information as well as our more up-to-date, often better, but not complete area-covering OSM data (barrier=kerb and residential landuse areas, which are often traditionally mapped to the kerb in Berlin). Theoretically, you could replace this data completely with "area:highway" OSM data, but we don't use them here yet (except in some intersection areas, see below).

Secondly, the line geometries of the parking lanes/parked cars are manually post-processed so that they fit exactly to the kerbs. Sorry, that I can't provide a magic receipe for that ;) With snapping, this can be automated well in QGIS, but to make it perfect and exclude "artefacts", I invested a few hours once to clean the data manually.

We now increasingly map accurate road width information in our area ("width:carriageway") and from various attributes (lanes, highway classification, oneway, number and location of parking and cycle lanes etc.) this information can also be interpolated for roads without any width attribute. I use this partly in the Python script for QGIS to generate the "basic version" of the parking lanes. But if you would render them on this basis, then unfortunately no such exact representation of the parking lanes would be possible as it is in the map - most parking lanes/cars would be a bit misaligned... (This could also be changed by mapping according to the "kerbside" scheme proposed by the makers of Curb LR: https://medium.com/sharedstreets/openstreetmap-and-curb-regulations-7812ee582a33 - but this is not yet used in OSM and the "parking:lane" scheme is so well established that it would probably not stand a chance).
Last week, the road markings and turn lanes have been added to the rendering of the micro map - these can be derived more or less well from the OSM data if lane attributes are mapped consistently (you know that, of cause). To improve the rendering, I have added "placement" tags in some places. In addition, intersection areas with "area:highway" + "junction=yes" are taken into account to "cut off" the markings in intersection areas. In return, I'm rendering "footway=crossing" ways there (or derive them from crossing nodes) to get more realistic crossing representation.
So a lot of this rendering only works because the area here is mostly mapped in a very detailed way... And unfortunately, the methodological requirements mentioned above mean that this micro-mapping map style cannot be easily extended to other areas or that it would require a generic lane model such as that of A/B Street (which can certainly come very close to reality if road attributes are mapped well). The generation of your data/the A/B Street lane model probably works completely differently and far more professionally, and therefore in a generic way (but as I said, I'm more an amateur and initially the map was only intended to visualise the parking data in our area...).
Regarding the second question: The topic of "public parking space" is currently being discussed very actively at all political and social levels in Berlin. However, there is a lack of empirical data on how much parking space there actually is - even more a lack of open data. We discovered that this data can be generated very precisely on the basis of OSM and the parking space analysis was therefore initially a little "tech demo" of how something like this can work on the basis of geodata. The response was great, some local initiatives use the data for political argumentation and planning and even two engineering companies, which do such data collection professionally, had questions, as they apparently do not yet work in such a "modern" way ;)
Lastly: Regarding A/B Street developments, I am always really impressed... "This is frankly the coolest thing I have seen in a long time" actually expresses pretty exactly what I feel when I see it ;) By the way, I love the Streetmix-like Road Editor approach - I've been dreaming of something like that for a very long time! It's also nice to see that the cycleway:separation scheme is taken into account ;)

(Sorry for the slow response)

I'm more an amateur and initially the map was only intended to visualise the parking data in our area

No need to be humble here -- even if you feel like a GIS beginner and this work would be hard to replicate elsewhere, it's still a fantastic map. I consider it a gold standard to build towards in more areas. I'm slowly puzzling through some ideas how to generalize the approach with less effort and in places without carriageway area data.
But actually, I was inspired to dig through Seattle's open data portal again, and found that there is pavement edge data (links here):

I think I've come across a few other examples of this in the US before. If the license is usable and we could devise some heuristic for chopping up the white negative space there into distinct road and junction polygons, then possibly we could propose importing these as area:highway polygons in some cities, with lots of human editing. And once we build a few demos of things using this kind of data, maybe mappers will be inspired to add more coverage and maintain it.
Another idea: integrate A/B Street's intersection geometry algorithm as a plugin to ID and JOSM. Then somebody can click a road center-line tagged in OSM, invoke the plugin, and auto-generate polygons for the road and junction. The result will be pretty bad in complex situations, but decent for many cases. The plugin could have a slider to adjust things like road width to match it to satellite easily. This could be another way to seed an area with more detailed area polygons.

I invested a few hours once to clean the data manually.

A lesson that I constantly forget is that automating GIS data is never a complete solution; cleaning things up manually is always necessary. I don't how many hours I've cumulatively spent trying to tune geometry algorithms to work well in a pretty small chunk of a city that I'm interested in. At some point, it feels like it'd be much more effective to instead develop simple plugins to more easily map highway areas in ID/JOSM. If the UI is easy and responsive, then drawing more detail against satellite -- in cases where tree cover doesn't make it impossible -- seems more plausible.

However, there is a lack of empirical data on how much parking space there actually is - even more a lack of open data.

I've seen similar debates in US cities. I'm curious, have you attempted to measure parking spaces associated with private buildings? Not just big commercial parking garages (though I don't remember seeing many of those in Berlin...), but the few spots tucked away behind an apartment building. Quantifying the number of spots is pretty hard, since people can get creative with packing vehicles into the space. But... somehow at least mapping the physical space would be interesting. But of course, much tougher than public street parking.

the few spots tucked away behind an apartment building

behind seems relatively easy but time consuming with decent aerial imagery (as in case of such areas getting turned into parkings they tend to eliminate any trees). And street parking mapping is also really, really time consuming anyway.
Underground parkings are much trickier.

The "private" parking spaces are also included in the evaluation to a (probably) very large extent. Some can be seen from the public space ("behind the fence"), others only become "visible" on aerial images. In Berlin, we are lucky that the city provides very good aerial images as OpenData every year, so that you can look under almost every tree using the images from different years ;) "Informal" parking spaces are not included in the analysis (as far as they are distinguishable from the official ones from remote).
And yeah, the underground parkings are a bit more complicated - partly their footprints are visible in our OpenData, but sometimes not. In some cases you can roughly derive their footprints from aerial imagery (according to the structures above the underground parking), in others you can just estimate ("somewhere below this building/behind this entrance"), or sometimes I was able to go in and count how many vehicles park there, got the numbers from the owners, from websites or plans. (Because this data is mostly not verifiable on the ground for the public, I didn't include them to OSM – only the entrances.)
In the end, with all the accuracy, one should keep in mind that the margin of error is rather small compared to the high total number of parked cars. So if some parking spaces are missing behind a building or the capacity of an underground garage is wrongly calculated, it doesn't make that much difference.


VCS: Remove project_path VCS property, let plugins decide

Screenshots:

On a completely new repo (no .git directory in current or parent paths)

On a seasoned repo (parent path has a .git directory)

This PR was made in parallel with - https://github.com/twaritwaikar/godot-git-plugin/pull/1
The project path is no longer required because if the plugin can self-identify the project directory, then there is no need for the user to customize this behaviour. Using a bad project path will result in frustrating errors since we expect the plugin to work the same way as the git CLI.

Can be cherrypicked to 3.x

Thanks!


Translation with multiple commas not readable

Hello, I have an issue using your library when I'm facing complex cron expressions like :
5-45/10,*/5,9 * * * *
The human translation is not understandable.
Expected : Every 10 minutes from 5th minutes to 45th minute, every 5 minutes, at 9th minute
Actual : Every 10,* minutes, minutes 5 through 45 past the hour
Thank you for your time.

@BilalMakhlouf - Thanks.  I'd welcome a pull request to fix this.

@bradymholt Have you had a chance to check out my pull request for this issue?

Followed up with Brady by email today.

I am fairly sure a segment with commas indicates multiple values that stand on their own and do not modify a prior value.
In ArchLinux cron documentation, it says a comma "List[s] multiple values separated by a comma."
So, I would expect 5-45/10,*/5,9 * * * * to output something similar to:

Every 10 minutes, minutes 5 through 45 past the hour, every 5 minutes, and at 9 minutes past the hour

So, there are 3 values within the segment:

5-45/10
*/5
9


Stop global event listener

what do you think?

Hi Max, you may want to select Agam as reviewer. You can assign that from the right pane by clicking the gear icon for reviewer. I saw you haven't made any changes in this PR other than your empty test file, so I assume it was your test PR to experiment.

uh, no that sounds wrong. i will see what agam says

thanks for telling me to assign someone, I didn't know that

@x5hemmax I meant to say you need to select @agamm as reviewer.

@agamm I have added the detachGlobalEventListener function and it seems like working in my test. Can you please review and approve the change.

Fixed the merge conflicts and tested it working.


Suggestion isn't included in quick fixes

Bug Report
Maybe this is by design, but I still find it a bit awkward. The issue is that the suggestion that appears when hovering over a TS error in VS Code (I could also reproduce it in VS, so it seems like a tsserver behavior and not an IDE thing) isn't included as a quick fix. See the code below for an example.
🔎 Search Terms

default
export

💻 Code
Say I have file1.ts with the following contents:
const foo = 0;

export default foo;

And I have the following in file2.ts:
import { foo } from './file1';

Because file1.ts has no named exports, the code in the second file has an error. When I hover over the squiggle, VS Code displays:

Despite the dialog mentioning exactly the fix that I would like to apply, that fix isn't included as a possible refactoring:

My expectation is that the suggestion mentioned in the hover dialog is included as a quick fix.

I actually think the suggestion is kind of a stretch unless the attempted named import matches the declared name of the default export or the filename from the other module:
// file1.ts
export default function foo() {}

// file2.ts
import { foo } from "./foo";   // you definitely meant to import default
import { file1 } from "./foo"; // same here
import { bar } from "./foo";   // no idea what you meant

We do generally aim to offer a code fix matching the suggestion in the error message, but I would actually tighten up the conditions where a default import is suggested, then add a code fix to match.


Sidebar not getting cleared after logout

Raised PR #4, This issue can be closed now

Looks good! Closing this issue.


correct number of qlat dataframe columns

In mc_reach.pyx, the required number of qlat dataframe column, , should be equal to or larger than total simulation time.   But, ![nsteps] in that line
Current behavior
Expected behavior
Steps to replicate behavior (include URLs)

Screenshots

I pushed a fix for this in #592.  The log is in fe6f8ff60cb8f7edf1e039f269f99a1043cb11dd, does that look more approriate?

@groutr @jameshalgren there has been some discussion recently around qts_subdivisions and what its intended use is and whether it is strictly required or if it can/should be essentially inferred from the input series.  Would either of you have any comments about this off the top of your head?  It is relevant here, but we can also move that to its own issue if we want, but this made me think of the disscussion.

Good question. We had the parameter there for early testing with un-registered input time series with no time metadata from which to infer the relationship to the simulation timesteps.
A related question that we wrestled with early on was whether to feed the array of qlateral values to the compute kernel already expanded out for all of the intermediate time steps or to keep it in the original hourly form and do the expansion in the kernel.


Partial scan | DFK72AUC02

Using your MS Software "IC Capture-2.4" it is possible to define an area which will be scanned with a higher frame-rate (e.g.: 2592x96 @ ~100fps).
Is this feature also available using tiscamera (tcambin/tcamsrc/v4l2-ctl/uvcdynctrl)?
We've tried to set it...
... as "capsfilter" for gstreamer:
gst-launch-1.0 --gst-debug=tcambin:5 tcambin ! video/x-bayer,width=2592,height=96,framerate=90/1,format=grbg ! bayer2rgb ! videoconvert ! autovideosink
...via tcam-ctrl
(after we fixed the set-function)
and v4l2-ctl --set-fmt-video command
but until now we were not able to get the partial scan working.
Could you give us an advice, please?
Thank you for any help you can provide!

Hello
v4l2 does not support variable resolutions, as DirectShow. Therefore, there is a limited list of resolutions available only in Linux. However, using lower resolutions leads to higher frame rates.
Using the the tcam-ctrl program you can see the available video formats and frame rates.
Also you can shorten your pipeline:
st-launch-1.0 --gst-debug=tcambin:5 tcambin ! video/x-raw, width=1024,height=768 ,framerate=60/1,format=RGB ! videoconvert ! autovideosink
The debayering is done in the tcambin.


Make sure that we don't pipe stdin when starting a job

Starting with version 13, ripgrep always stats stdin and if it's not a TTY it uses it to read data. Unfortunately, Neovim always attaches a pipe to stdin by default and that leads to ripgrep reading nothing and it essentially breaks vim-grepper when targeting a recent version of ripgrep.
(see https://github.com/BurntSushi/ripgrep/issues/1892 for more info)
This was fixed in nvim by adding an option to jobstart to not pipe stdin (see https://github.com/neovim/neovim/pull/14812). So we use this here which I verified fixes search through rg.
Note that I'm not 100% sure how to gate this. It was technically added as a commit to neovim after 0.5 shipped so I imagine it will technically be released in 0.5.1. But it should also be harmless to only gate this to nvim since the options are a dictionary and old versions of neovim will just ignore that stdin key.
This fixes https://github.com/mhinz/vim-grepper/issues/244

Great, thanks again.
Let's wait for @mhinz to approve/merge it (I cannot).

I have merge rights, just one question, will if has('nvim-0.5.1') also work with later versions?

Yes it will. I’m running the nightly 0.6 version and it works fine.


在严格StrictMode模式下报错

[ ] I'm sure this does not appear in the issue list of the repository

Basic Info

Package Name And Version<EMAIL_ADDRESS>Browser: chrome101.0.4951.41

Extra info
Warning: findDOMNode is deprecated in StrictMode. findDOMNode was passed an instance of Transition which is inside StrictMode. Instead, add a ref directly to the element you want to reference. Learn more about using refs safely here: https://reactjs.org/link/strict-mode-find-node
What is expected?
Warning: findDOMNode is deprecated in StrictMode. findDOMNode was passed an instance of Transition which is inside StrictMode. Instead, add a ref directly to the element you want to reference. Learn more about using refs safely here: https://reactjs.org/link/strict-mode-find-node
Steps to reproduce
Warning: findDOMNode is deprecated in StrictMode. findDOMNode was passed an instance of Transition which is inside StrictMode. Instead, add a ref directly to the element you want to reference. Learn more about using refs safely here: https://reactjs.org/link/strict-mode-find-node

你好，目前没有很好的办法，官方提供的 ref 无法覆盖类组件的情况。我们会继续跟进这个问题。

Duplicate: https://github.com/arco-design/arco-design/issues/875

覆盖不到类组件, 意思就是类组件下仍然有提示, 但hooks下没有提示
这很好啊, 不太理解为什么会成为一个理由呢


Newly Added: BC, ADIH, BWL

Closes #329
What new changes did you make? Tick all applicable boxes

[x] Added new abbreviation
[ ] Fixed something in the source code
[ ] Added a new feature
[ ] Fixed the docs (README.md, CONTRIBTUING.md, etc)

Describe the new changes you added.
BC - Because
ADIH - Another DAy In Hell
BWL - Busting With Laughter

Share a screenshot of new changes
N/A

Bursting! Fixed spelling now :)


🔧 'react/bridging/CallbackWrapper.h' file not found

How were you trying to build the app?
I tried to build my app with XCode
I tried troubleshooting step no 5:
Make sure you have created a Swift bridging header in your project
However, I'm not prompted by the XCode like guide suggests:
Press Create Bridging Header when promted.
Full build logs
/Users/user/Library/Developer/Xcode/DerivedData/mobile-apywzqgnyufposbigwaniblslult/Build/Products/Debug-iphonesimulator/ReactCommon/ReactCommon.framework/Headers/react/nativemodule/core/ReactCommon/CallbackWrapper.h:11:10 'react/bridging/CallbackWrapper.h' file not found

Project dependencies
"dependencies": {
    "@react-native-community/netinfo": "^11.2.1",
    "@react-native-firebase/analytics": "^18.7.2",
    "@react-native-firebase/app": "^18.7.2",
    "@react-navigation/native": "^6.1.9",
    "@react-navigation/native-stack": "^6.9.17",
    "@reduxjs/toolkit": "^2.2.0",
    "@sayem314/react-native-keep-awake": "^1.2.2",
    "axios": "^1.5.1",
    "events": "^3.3.0",
    "jwt-decode": "^3.1.2",
    "lottie-react-native": "^6.5.1",
    "ramda": "^0.29.1",
    "react": "18.2.0",
    "react-native": "0.72.3",
    "react-native-audio-recorder-player": "^3.6.1",
    "react-native-bouncy-checkbox": "^3.0.7",
    "react-native-circular-progress-indicator": "^4.4.2",
    "react-native-config": "^1.5.1",
    "react-native-device-info": "^10.12.0",
    "react-native-file-logger": "^0.4.1",
    "react-native-fs": "^2.20.0",
    "react-native-inset-shadow": "^1.0.3",
    "react-native-keyboard-aware-scroll-view": "^0.9.5",
    "react-native-reanimated": "^3.5.4",
    "react-native-render-html": "^6.3.4",
    "react-native-safe-area-context": "^4.7.4",
    "react-native-screens": "^3.27.0",
    "react-native-sound": "^0.11.2",
    "react-native-svg": "^13.14.0",
    "react-native-vision-camera": "^3.9.0",
    "react-native-zip-archive": "^6.1.0",
    "react-redux": "^9.1.0",
    "rn-fetch-blob": "^0.12.0",
    "react-native-vision-camera-face-detector": "^1.2.2",
    "react-native-worklets-core": "^0.2.4"
  }

VisionCamera Version
3.9.0
Target platforms
iOS
Operating system
MacOS
Can you build the VisionCamera Example app?
Yes, I can successfully build the Example app here
Additional information

[ ] I am using Expo
[X] I have enabled Frame Processors (react-native-worklets-core)
[X] I have read the Troubleshooting Guide
[X] I agree to follow this project's Code of Conduct
[X] I searched for similar issues in this repository and found none.

Hi @mikoprus have you checked #1121

adding s.dependency "ReactCommon" to VisionCamera.podspec fixes the bug

Great, maybe I need to add that again.

@mrousavy I've created a PR adding this: https://github.com/mrousavy/react-native-vision-camera/pull/2620


SSO Middleware

As a stakeholder
I want to have SSO capabilties
So that users can easily sign up and login to the application

[ ] Set up session middleware
[ ] sign up for keycloak thru platform services
[ ] create login view
[ ] create login/logout buttons
[ ] test and ensure proper login/logout functionality

we already have the SSOs


Update Google Cloud Storage documentation and add Azure Blob Storage example

While I was going through the 🤗 Datasets documentation of the Cloud storage filesystems at https://huggingface.co/docs/datasets/filesystems, I realized that the Google Cloud Storage documentation could be improved e.g. bullet point says "Load your dataset" when the actual call was to "Save your dataset", in-line code comment was mentioning "s3 bucket" instead of "gcs bucket", and some more in-line comments could be included.
Also, I think that mixing Google Cloud Storage documentation with AWS S3's one was a little bit confusing, so I moved all those to the end of the document under an h2 tab named "Other filesystems", with an h3 for "Google Cloud Storage".
Besides that, I was currently working with Azure Blob Storage and found out that the URL to adlfs was common for both filesystems Azure Blob Storage and Azure DataLake Storage, as well as the URL, which was updated even though the redirect was working fine, so I decided to group those under the same row in the column of supported filesystems.
And took also the change to add a small documentation entry as for Google Cloud Storage but for Azure Blob Storage, as I assume that AWS S3, GCP Cloud Storage, and Azure Blob Storage, are the most used cloud storage providers.
Let me know if you're OK with these changes, or whether you want me to roll back some of those! :hugs:

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

Hi @stevhliu, I've kept the >>> before all the in-line code comments as it was done like that in the default S3 example that was already there, I assume that it's done like that just for readiness, let me know whether we should remove the >>> in the Python blocks before the in-line code comments or keep them.

Comments are ignored by doctest, so I think we can remove the >>> :)

Cool I'll remove those now 👍🏻

Sure @lhoestq, I just kept that structure as that was the more similar one to the one that was already there, but we can go with that approach, just let me know whether I should change the headers so as to leave all those providers in the same level (h2). Thanks!


modify /edit

Allow multiple fields of expense to be edited. Currently have to call /edit for each field to be edited (category, date and amount. Have to call "/edit" three times if we want to modify all three)

Taking up this issue

Took this up as our discussion earlier today.


Support Unicode Symbols

This is the third in a series of PRs to give KaTeX the ability to recognize Unicode character input. This one supports some but not all letter-like symbols and some other symbols. All the characters in this PR will produce textord atoms in math mode.
The list is not comprehensive. It’s a bit eclectic. Some of these characters are important. Others were just easy to include. A few important characters are omitted because it wasn’t clear to me which is the best way to implement them. I’ll address those characters in future PRs.

Double-struck characters cannot be done with current fonts by just setting a flag in symbols.js. Take ℂ, for example. In the katex-ams font, the character that looks like ℂ is not at code point U+2102 but is instead at code point U+0043. So, the symbols.js flag technique doesn’t work. Rather than wait for font revisions, I wrote macros.

As in PR #933, the following table shows the KaTeX function to which each character is mapped and confirmation whether unicode math and/or John Cook’s conversion page map it the same way.

I think that's sensible.  I think revising the fonts would introduce other complications as it relates to handling font commands such as \mathbb, etc.

Let's hold off on this until after PR #973 lands. I'll revise this PR accordingly.

The most recent commit reverts a RegEx in MacroExpander.js. So the code in this PR is now dependent on PR #973.

Now that PR #973 has landed, this should be good to go.


Add scoreboard total and maxPoints and fix scoreboard bugs

Resolves #48
The scoreboard now uses the number of teams instead of pages, that way pagination can be done client-side.
I added two fields to the scoreboard:

total - the total number of teams in the response (max 100)
maxPoints - the maximum points of all challenges as of the moment

I also fixed bugs with representation, validation, etc. of scoreboard requests.

(yes I know the things say WIP and untested, that was before I tested them, they work)

@chop0 can you make sure this is good? I'll take care of any conflicts on merge

woops forgot to merge :sweat_smile:


Question: How to handle different server responses in farfetched?

In this repository, the discussions section is disabled. I believe my question should be located there.
I would like to see some API like this but maybe farfetched has a different approach to solving the same case.
import { createJsonQuery } from '@farfetched/core'
import { runtypeContract } from '@farfetched/runtypes';
import { Record, Literal } from 'runtypes';

const NotFoundUserResponse = Record({
  result: Literal(false),
  error: Record({
    type: Literal('USER_NOT_FOUND'),
   })
});
const SuccessUserResponse = Record({
  data: User,
  result: Literal(true)
});

const query = createJsonQuery({
  // other options...
  contracts: {
    notFound: runtypeContract(NotFoundUserResponse),
    success: runtypeContract(SuccessUserResponse),
  }
});

sample({
  clock: query.finished.notFound,
  target: navigateUserToHomePageFx,
});

sample({
  clock: query.finished.success,
  // do something when everthing is ok
});

sample({
 clock: query.finished.unexpected,
 // do something when server response type is unexpected
});

Hello! Thanks for the idea 🙏 I'll think about it, let's stick it as a feature request for now.


support prettyPrint in browser display

Small request :-)
We are not using prettyPrint but do support a browser friendly display (via ResponseHighlighterInterceptor).
I would like the generated html do include a "view as pretty print" option, adding a _pretty=true parameter to the url.

Hi Allan,
Maybe I'm missing something obvious, but why would you want this? The ResponseHighlighterInterceptor already pretty prints by default (even if that parameter isn't found in the URL) so wouldn't this link not have any effect?

Hi James. I do not experience a pretty print by default, so adding ?_pretty=true makes a difference.
E.g.. http://username:password@my-environment.com/Organization/OrgId?_pretty=true.
I am using HAPI 1.6. I also notice that I do not have the "Response generated in "-display that is shown on http://fhirtest.uhn.ca/baseDstu2/Patient/.

Well I'll be.
I always thought pretty was the default in that interceptor, but you're right, it isn't. I guess I've always used it on servers where pretty was the default all over! Learn something new every day..
I see no reason why it shouldn't be the default when using the interceptor.. Pretty coloured source that's all on one line is not very useful :)
Will fix.


[EventHubs] Auth consolidation

Fix for issue #8603

Removed legacy auth code
Removed explicit SAS token auth
Updated SharedCredentialAuth to align to TokenCredential protocol

Can one of the admins verify this patch?

/azp run python - eventhubs - tests

/azp run python - eventhubs - tests

/azp run python - eventhubs - tests

/azp run python - eventhubs - tests


feat(apollo): Create removeExercise resolver

Motivation
Currently, there is no way to set the removed flag on an exercise when an admin wants to remove it. The removed flag is a field in the database that indicates whether an exercise has been removed or not.
The PR is adding a resolver that will allow an admin to set the removed flag to true when they want to remove an exercise. This will mark the exercise as removed in the database, and it will no longer be visible to users.
Steps taken

Created the resolver removeExercise
Ensure everything is working as intended by adding tests for the resolver
Created the query file for removeExercise resolver so it can be easily used in the client-side
Update GraphQL index.ts with the new resolver and hooks.

Unrelated steps

Fixes some typos

Resolver logic

It receives one argument, the user id.
It fetches the exercise to be removed from the database using the provided id.
It retrieves the id of the user making the request (authorId) from the request context (req).
If the user is not an admin and the authorId does not match the authorId of the exercise being removed, the resolver throws an error, indicating that the user is not authorized to remove the exercise.
If the exercise is already removed, the resolver throws an error, indicating that exercise is already flagged.
If the user is authorized to remove the exercise, the resolver updates the exercise in the database, setting the removed field to true. It also includes the author and module relations in the update.
The resolver returns the updated exercise.

Testing results
Executes removeExercise:
mutation {
  removeExercise(id: 1) {
    id
  }
}

Result:
{
  "data": {
    "removeExercise": {
      "id": 1
    }
  }
}

Result if it's already removed:
{
  "errors": [
    {
      "message": "Exercise is already removed",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
  "data": null
}

Testing instructions

Go to /api/graphql
Run the queries from Testing result section
Make sure they return the correct results

Related issues

#2630

This is out of the scope of this particular PR but related, maybe removed should be removedAt/removed_at as well like some other booleans we have changed to date | null before.
Like flagged at and the user who flagged it, it would make sense for removed to be like this too with removed at and removed by data.
And even if removed remains a boolean it should not be an optional field, it should either be required with false as default so only two values are possible, or it should be a date field with default value being date. Currently removed can hold three values null, false, true.

This is out of the scope of this particular PR but related, maybe removed should be removedAt/removed_at as well like some other booleans we have changed to date | null before. Like flagged at and the user who flagged it, it would make sense for removed to be like this too with removed at and removed by data. And even if removed remains a boolean it should not be an optional field, it should either be required with false as default so only two values are possible, or it should be a date field with default value being date. Currently removed can hold three values null, false, true. My suggestion would be to refactor the schema for exercise and improve it with regards to remove before this PR.

I agree. It makes sense.


URLParts -> NSURLComponents

I think you'll have slightly less janky code if you use NSURLComponents to digest a particular url into its components rather than scanning through strings for separator characters.

IIRC there was a reason I didn't, but I can't remember what it was now.

Oh, right, it's because NSURLComponents chokes on non-ASCII like NSURL does. :(

Closing for afformentioned reason; may revisit in the future.


[v6][Bug] Table headings have different font-size in header and footer

Reported by Pedro: Column titles use different sizes in header and in footer. I think we should use the footer size on header too.

Fixed by https://github.com/Laravel-Backpack/theme-tabler/pull/102


proposal: type and expression super parameters

Introduction of the proposal
This is not a generic proposal (at least not a generic proposal mainly), though it may solve some problems caused by lack of generics.
An expr super parameter means a lazy-evaluated value. For example, in the following function, b and c will only be evaluated  if a is evaluated to true.
func Log(a bool, b string:expr, c ...interface{}:expr) {
	if a {
		fmt.Printf(b, c...)
	}
}

An example using type super parameters:
func make(T Map|Slice|Channel:type, sizes ...int) T {
	switch T.kind {
	case Map:
		return makeMap(T)
	case Slice:
		return makeSlice(T)
	case Channel:
		return makeChan(T)
	}

	panic("invalid kind")
}

func new(T Any:type) *T {
	var t T
	return &t
}

func Add(T String|Numeric:type, a, b T) T {
	return a+b
}

var x = Add(int, 1, 2)
var y = Add(String, "Go", "lang")
var z = Add(float32, 1.23, 7.89)

A type super parameter must be an instance of a kind. The Map, Slice, Channel, String, Numeric and Any used in the above example are some pre-declared kind identifiers. We can also declare custom kinds:
kind MyKind Array|Slice
kind Unsigned Uint8|Uint16|Uint32|Uint64|Uintptr

kind is a new keyword. (In fact, there are not many variations of custom kinds. So this keyword is not very essential.)
In the parameter list, type super parameters must be declared in front of other parameters.
We can view the current general parameter declaration form v T as a simplified form of v T:var.
The syntax is not the core of this proposal. Any improvement ideas are welcome.
Benefits of this proposal
type super parameters are mainly to write some simple custom generic functions.
expr super parameters are mainly to use some functions in a clean but still performant way. For example, when using the glog package, the clean way
glog.V(2).Info(a + b + c + d)

is less efficient than the verbose way
if glog.V(2) { glog.Info(a + b + c + d) }

where a + b + c + d is a string concatenation expression.
The current declaration of the Info method:
type Verbose bool

func (v Verbose) Info(args ...interface{}) {
	if v {
		logging.print(infoLog, args...)
	}
}

By using expr super parameters, we can declare it as
func (v Verbose) Info(args ...interface{}:expr) {
	if v {
		logging.print(infoLog, args...)
	}
}

then the above clean way will be as efficient as the verbose way.

Please note that, the "lazy-evaluated" mentioned above means the operations in the said expressions are lazy evaluated, not mean the elementary values in the AST trees of the expressions.
For example, the following program prints 7 (1 + 2*3)  instead of 79 (7 + 8*9).
package main

func foo(v int:expr, c <-chan int) int {
	<-c
	return v
}

func bar(x, y int) int {
	return x * y
}

func main() {
    var a, b, c = 1, 2, 3
    var d = make(chain struct{})
    go func() {
    	a, b, c = 7, 8, 9
    	close(d)
    }()
    println(a + bar(b, c), d) // 7
}

Maybe, to keep the lazy evaluated results as the same as if they are not lazy evaluated

function calls will never be lazy evaluated.
pointer dereferences, selectors, index expressions will never be lazy evaluated.

t looks the troubles brought by expr parameters are more than their benefits.
String concatenation is only situation I am aware of  which needs to be lazy evaulated. Could compiler optimizations help here? If the answer is yes, I will split this proposal into two.


Fix remove playing notification when app is swiped from recents

Fix GitHub issue #250
When I swipe UAMP away from recents, the removeNowPlayingNotification() isn't called because in the method updateNotification() the metadata is already null and this check returns without removing notification if(mediaController.metadata == null) return . So because of this notification is stuck in the notification tray and control icons also does not work.

Thank you very much for the PR!
It'll probably take a bit of time for me to merge it, but I just wanted to let you know I saw it and it's on my list of things to do!

Thanks again!


How to run ./deploy.sh as in getting python version error

Hi,
When running ./deploy.sh in google cloud shell. I'm getting below error
"This script requires Python 3.8 or higher!
You are using Python 3.7."
How to update cloud shell to 3.7 or I have to run this script somewhere else?

Hello,
Iris3 code uses some Python 3.8 features and so requires that.
I have been using my laptop rather than CloudShell, as CloudShell is ephemeral, but you could still do that.
Pyenv is one option : https://stackoverflow.com/a/57403582/39242
Or building   Python3.8  as follows (replace 3.7 with 3.8) https://stackoverflow.com/a/54347468/39242

Closing this given the solution above


Automatic Reconciliation

@kgeographer:
"...the Digging Into Early Colonial Mexico (DECM) dataset I long held off preparing for my friends Paty Murrieta-Flores and Bruno Martins is uploaded, and a reconciliation task initiated.
"When I began to perform matching on the hits, I saw that a great many are so close in location, with exact or partial name matches, that this makes a great candidate exemplar for some work I think? is on the relatively near term agenda for WHG - namely, improvement of the reconciliation process such that highly likely matches like that go into the "pass0" queue, or another similar queue that can be "auto-accepted" to save marching through and confirming hundreds of obvious hits. Long been on the wishlist of ways to make a better/easier experience of recon/matching/accessioning."

Will be addressed by https://github.com/WorldHistoricalGazetteer/whg3/issues/381


[Feature Request] - kubectl "protocol"

Description
It would be really cool to be able to browse and copy files to/from services running in kubernetes using termscp. I imagine an initial remote screen could be a listing of namespaces, and when you drill in you can see pods or services running in that namespace, and then you can navigate the file system like normal.
Changes
A kubectl "protocol" would need to be added to termscp.
Implementation
You can copy files to and from pods running on kubernetes like such:
kubectl cp --namespace prod ./local/path pod-name:./remote/path

Remote files can be listed like such:
kubectl exec --namespace prod --stdin --tty pod-name -- ls ./remote/path

Namespaces can be listed as such:
kubectl get namespaces

Using these (and other) commands it should be possible to implement a kubectl protocol to be able to browse and copy files to/from services running in kubernetes.

I currently use navi to do very similar things for my running services, just with fuzzy-finding rather than a TUI: https://github.com/knpwrs/dotfiles/blob/f08642f35fe224949ad0c3a959dbe1d3ec24e6f4/home/config/navi/k8s.cheat

So the implementation is done and it'll be released along with 0.14 very soon. Unfortunately, it won't have the possibility to display namespaces, pods and containers atm.
This would require a change in remotefs-kube to have a side client to work with that, which would be cool, but needs to be implemented. So expect this change in a 0.15.


Fix/cleanup effects

This is a small change that adds a cleanup to two useEffect calls. It prevents possible memory leaks.

I'm confused. How is
function myEffect() {
  const callback = generateCallback()
  return () => callback()
}

any different from
function myEffect() {
  return generateCallback()
}

??
Isn't that basically what you're doing in this PR? registerField() already returns the cleanup function.

Ah I see how it works now, you're right, it's already doing it.