7.4.5 windows-arm build fails to start with TypeInitializationException

Prerequisites

[X] Write a descriptive title.
[X] Make sure you are able to repro it on the latest released version
[X] Search the existing issues.
[X] Refer to the FAQ.
[X] Refer to Differences between Windows PowerShell 5.1 and PowerShell.

Steps to reproduce
Unhandled exception. System.TypeInitializationException: The type initializer for 'System.Management.Automation.Configuration.PowerShellConfig' threw an exception.
---> System.BadImageFormatException: Could not load file or assembly 'C:\Program Files\PowerShell\7\Newtonsoft.Json.dll'. Format of the executable (.exe) or library (.dll) is invalid.
File name: 'C:\Program Files\PowerShell\7\Newtonsoft.Json.dll'
at System.Management.Automation.Configuration.PowerShellConfig..ctor()
at System.Management.Automation.Configuration.PowerShellConfig..cctor()
--- End of inner exception stack trace ---
at System.Management.Automation.Utils.GetPolicySettingFromConfigFile[T](ConfigScope[] preferenceOrder)
at System.Management.Automation.Utils.GetPolicySetting[T](ConfigScope[] preferenceOrder)
at Microsoft.PowerShell.CommandLineParameterParser.GetConfigurationNameFromGroupPolicy()
at Microsoft.PowerShell.ConsoleHost.ParseCommandLine(String[] args)
at Microsoft.PowerShell.UnmanagedPSEntry.Start(String[] args, Int32 argc)
at Microsoft.PowerShell.ManagedPSEntry.Main(String[] args)
[process exited with code<PHONE_NUMBER> (0xe0434352)]
You can now close this terminal with Ctrl+D, or press Enter to restart.
Expected behavior
Powershell 7 (Arm) starts.

Actual behavior
Crashes

Error details
No response
Environment data
Cannot access

Visuals
No response

Duplicate of https://github.com/PowerShell/PowerShell/issues/23969, but I think it should either support both versions installed side-by-side, or make the installer prevent both.

If this is a Microsoft update problem, then that's something the team can investigate.  As noted, if this is installing multiple architectures on same system, then that is not supported.  Use the zip package if you are a developer that requires both architectures.

Still bad UX

Still bad UX

It is not problem unique to PowerShell. See MSI for ARM64 Java should install in C:\Program Files, not C:\Program Files (Arm). The mistake referenced with Java was they were installing ARM64 into a directory for 32-bit ARM.
This dates back to the move from WIN32 to WIN64, and then porting to DEC-Alpha,  MIPS, PowerPC, ARM and more recently to ARM64.
The decision was made that "C:\Program Files" is allocated to the primary architecture of the host. However, for backwards compatibility it is possible to install x64 programs on non-x64 64-bit machines.
So on ARM64 we have
C:\Program Files and C:\Windows\System32
C:\Program Files (Arm) and C:\Windows\SysArm32
C:\Program Files (x86) and C:\Windows\SysWOW64
The choice could have been made that when an MSI installs x64 onto an ARM64 machine then it goes into "c:\Program Files (x64)" rather than "C:\Program Files", but they didn't.

My point is that this kind of corruption is avoidable at the installer
level. .msi custom actions (type 2) can run executables, and checking the
architecture of a .exe or .dll is maybe 4 lines of Rust. Might add a couple
hundred kb at most?
I'd be suprised if there isn't already a recipie for making installers
resilient, but it's hardly much work to add a check and prevent this kind
of corruption, and stave off the inevitable influx of support cost.
On Sat, Sep 14, 2024, 11:43 AM rhubarb-geek-nz @.***>
wrote:

Still bad UX
It is not problem unique to PowerShell. See MSI for ARM64 Java should
install in C:\Program Files, not C:\Program Files (Arm)
https://github.com/microsoft/openjdk/issues/344. The mistake referenced
with Java was they were installing ARM64 into a directory for 32-bit ARM.
This dates back to the move from WIN32 to WIN64, and then porting to
DEC-Alpha, MIPS, PowerPC, ARM and more recently to ARM64.
The decision was made that "C:\Program Files" is allocated to the primary
architecture of the host. However, for backwards compatibility it is
possible to install x64 programs on non-x64 64-bit machines.
So on ARM64 we have
C:\Program Files and C:\Windows\System32
C:\Program Files (Arm) and C:\Windows\SysArm32
C:\Program Files (x86) and C:\Windows\SysWOW64
The choice could have been made that when an MSI installs x64 onto an
ARM64 machine then it goes into "c:\Program Files (x64)" rather than
"C:\Program Files", but they didn't.
—
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/24226#issuecomment-2351037420,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA2LH53BX6TQXGG65ROB3TZWRKQRAVCNFSM6AAAAABNKGEGGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRGAZTONBSGA
.
You are receiving this because you authored the thread.Message ID:
@.***>

The ARM64 MSI is a very new feature for PowerShell despite ARM64 windows having been available for a couple of years. The ARM64 MSI for Windows first came out with 7.4.3
An alternative would be to use WIX_NATIVE_MACHINE (example) to prevent the x64 being installed on ARM64.

On Sun, Sep 15, 2024, 6:34 AM rhubarb-geek-nz @.***>
wrote:

The ARM64 MSI is a very new feature for PowerShell despite ARM64 windows
having been available for a couple of years. The ARM64 MSI for Windows
first came out with 7.4.3
An alternative would be to use WIX_NATIVE_MACHINE (example
https://github.com/dotnet/aspnetcore/pull/37318) to prevent the x64
being installed on ARM64.
—
Reply to this email directly, view it on GitHub
https://github.com/PowerShell/PowerShell/issues/24226#issuecomment-2351526828,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAA2LH6RI4VTI3MWIX2IORDZWVPCHAVCNFSM6AAAAABNKGEGGCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNJRGUZDMOBSHA
.
You are receiving this because you authored the thread.Message ID:
@.***>

In Windows a process runs with a single architecture, and all native dlls loaded into that process must be of the same architecture.  The architecture of that process is governed by the exe program that is used by CreateProcess, which may be specified exactly or found on the path. Which dlls are loaded are then dependent on the exe and the path. DotNet's runtime permits 'any' architecture dlls to be used by any CPU.

An option would be to use WIX_NATIVE_MACHINE (example) to prevent the x64 being installed on ARM64.

Seems like we could use that variable to error out if the architecture doesn't match.  Marking this up-for-grabs if someone wants to make a fix.

Do we want to prevent x64 from installing on woa? That might have unintended consequences and will probably break a lot of tooling that installs pwsh automatically.

I think the correct thing to do is require uninstallation of x86_64 before allowing aarch64 to be installed and vice versa (which probably fixes update issues too, not just user error)
Only allowing the native version of a product limits workarounds for architecture specific issues.
I realize checking the currently installed architecture adds complexity, but if it's the most correct thing to do, we should probably do it, and open-source the approach used for the msi, so other products can do the same.
I could probably make a tiny rust executable with no dependencies (other than the ucrt), if that is the best way forward. I've had to implement this kind of thing in C# before, due to how bad .NET 4.X is about native/arch-specific dependency resolution.

Can I suggest testing aspnetcore and confirm the mechanism it claims to implement works?
I see PowerShell as part of the same family as aspnetnet & dotnet-sdk so it would be strange to adopt a different approach to the same problem.


Aim to base helloSystem on supported FreeBSD 12.2-RELEASE-p4 or 13.0-RELEASE

DRM
https://github.com/helloSystem/ISO/issues/1
From https://old.reddit.com/r/freebsd/comments/lgjuab/-/gn3t3qi/?context=1 for three relevant packages from latest:
grahamperrin@freebsd:/usr/home/grahamperrin $ su -
root@freebsd:~ # kldload /boot/modules/i915kms.ko
root@freebsd:~ # kldstat
Id Refs Address                Size Name
1   37 0xffffffff80200000  227ae70 kernel
2    1 0xffffffff8247b000   667e20 openzfs.ko
3    1 0xffffffff83011000     2698 intpm.ko
4    1 0xffffffff83014000      b40 smbus.ko
5    1 0xffffffff83015000    2eb93 vboxguest.ko
6    1 0xffffffff83044000     88c0 tmpfs.ko
7    1 0xffffffff8304d000   12bf6c i915kms.ko
8    1 0xffffffff83179000    75e10 drm.ko
9    4 0xffffffff831ef000    12d30 linuxkpi.ko
10    3 0xffffffff83202000    13f30 linuxkpi_gplv2.ko
11    2 0xffffffff83216000      6d0 debugfs.ko
root@freebsd:~ # pkg query '%o %v %R' drm-kmod drm-fbsd12.0-kmod gpu-firmware-kmod
graphics/drm-kmod g20190710_1 FreeBSD
graphics/drm-fbsd12.0-kmod 4.16.g20201016_1 FreeBSD
graphics/gpu-firmware-kmod g20201213 FreeBSD
root@freebsd:~ # date ; uptime ; uname -a
Fri Feb 12 21:40:02 UTC 2021
9:40PM  up 4 mins, 2 users, load averages: 0.29, 0.32, 0.15
FreeBSD freebsd 12.2-RELEASE-p3 FreeBSD 12.2-RELEASE-p3 GENERIC  amd64
root@freebsd:~ # freebsd-version -kru
12.2-RELEASE-p3
12.2-RELEASE-p3
12.2-RELEASE-p3
root@freebsd:~ # grep url /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
root@freebsd:~ #

success.

OpenZFS
From https://github.com/helloSystem/ISO/issues/125#issuecomment-774700352:

… it's reasonable to assume that this build will include sysutils/openzfs-kmod.

FreeBSD 12.2-RELEASE-p3 with the kernel module built from a 12.2 jail for quarterly:
https://user-images.githubusercontent.com/192271/107828490-e29b4480-6d80-11eb-88eb-ee3f7a796fe3.mp4

success.

ksnip
https://github.com/helloSystem/ISO/issues/17 (2020-11-21)
https://github.com/helloSystem/ISO/blob/9c09b32a5862fd31c7d5e584b66db9249756b090/settings/packages.hello#L68
A fixed version in quarterly:
grahamperrin@freebsd:/usr/home/grahamperrin $ pkg search ksnip
ksnip-1.7.3_1                  Screenshot and annotation tool
grahamperrin@freebsd:/usr/home/grahamperrin $ grep url /etc/pkg/FreeBSD.conf
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
grahamperrin@freebsd:/usr/home/grahamperrin $

https://www.freshports.org/graphics/ksnip#history 1.7.3_1 committed 2020-12-30.

Keep in mind that we cannot use quarterly packages because packages can disappear there from one day to the next. And we can only switch to 12.2 once we have a compatible Intel DRM package. Which means (if I am not mistaken) that we can only switch to 12.2 once the Intel DRM package in last quarter's packages is compatible with 12.2.
So, at least another quarter to wait?

… packages can disappear … from one day to the next, …

Disappearances should be infrequent (and annoyances should be avoidable). Maybe raise a separate issue for this, for a focused investigation that's not scattered across multiple issues.

… compatible Intel DRM package. …

Above, kldload /boot/modules/i915kms.ko – is this kernel module not what's required?

Above, kldload /boot/modules/i915kms.ko – is this kernel module not what's required?

I think it is - but does it work? https://github.com/helloSystem/ISO/issues/1

File systems support
From https://github.com/helloSystem/docs/commit/54147195c7c96095fc2e3a656be3acfbd829d147:

exFAT (Windows) once available in quarterly packages

– and:

EXT4 (Linux) once available in quarterly packages

https://github.com/vermaden/automount describes sysutils/fusefs-ext4fuse, https://www.freshports.org/sysutils/fusefs-ext4fuse/#history that's removed. Instead: sysutils/fusefs-ext2.
Available, logged:

http://beefy2.nyi.freebsd.org/data/122amd64-quarterly/563981/logs/exfat-utils-1.3.0.log
http://beefy2.nyi.freebsd.org/data/122amd64-quarterly/563981/logs/fusefs-exfat-1.3.0.log
http://beefy2.nyi.freebsd.org/data/122amd64-quarterly/563981/logs/fusefs-ext2-0.0.10_2.log

https://github.com/helloSystem/ISO/commit/f73b9d6bf78d0fbf04e9ae74f25ac74bcacea0d9 hello-0.5.0_0E5-FreeBSD-12.2-amd64.iso pre-release currently at https://github.com/helloSystem/ISO/releases/tag/experimental-12.2 is:

12.2-RELEASE
not 12.2-RELEASE-p3

@probonopd I suspect, no value in having hello-0.5.0_0E5-FreeBSD-12.2-amd64.iso alongside hello-0.5.0_0E5-FreeBSD-12.1-amd64.iso …
… consider withdrawing hello-0.5.0_0E5-FreeBSD-12.2-amd64.iso.
Thanks

For me the question is which FreeBSD version the next helloSystem version, 0.5.0 (~March 2021) should be based on. The experimental builds right now are building towards the upcoming 0.5.0. (The mid-term question will be whether 1.0.0 will be 13-based...).
I must have overlooked the difference between RELEASE and RELEASE-p3 (still leearning FreeBSD versioning). What I am looking for is "the latest security-patched RELEASE in the 12 series" - ideally without having to manually track p1, p2,... (is there a wy to say "latest 12 RELEASE"?)
And the question is, can any 12 later than 12.2 currently work properly (including Intel GPU drivers) with last quarter's packages (because quarterly seems to have missing packages from one day to the next)?

freebsd-12-2-release-p3-amd64 does not exist as an instance name on Cirrus CI (which is using Google Cloud Computing Engine instances).

freebsd-12-2-release-p3-amd64 does not exist as an instance name on Cirrus CI …

helloSystem documentation refers to https://cirrus-ci.com/ but it's somewhat mysterious.
How can I tell what does exist?

https://github.com/helloSystem/ISO/issues/135#issuecomment-780735139

say "latest 12 RELEASE"

freebsd-update upgrade -r 12.2-RELEASE

begins an upgrade to the latest version of 12.2-RELEASE, which is currently patch level 3 i.e. 12.2-RELEASE-p3.

Patch levels and security advisories
Thanks to debdrup in freenode (@debdrup in GitHub?) I recently learnt of this unnoficial page:

FreeBSD Patch Level Table

Under https://bokut.in/freebsd-patch-level-table/#releng/12.2

we have 12.2-RELEASE-p2 with reference to its one related SA
we do not yet have 12.2-RELEASE-p3 with reference to its two related SAs.

Posted to Reddit, unfortunately neither post was allowed:

https://www.reddit.com/r/freebsd/comments/l92s5g/-/
https://www.reddit.com/r/freebsd/comments/l92r3p/-/

… can any 12 later than 12.2 currently work properly … with last quarter's packages …

It's simple to test this for yourself with hello-0.5.0_0E6-FreeBSD-12.1-amd64.iso and VirtualBox. I'll make a screen recording to help you …

It's simple

Unfortunately not. This is a show-stopper to reasonable testing:

This means that we really need to find a way to make quarterly work for us, rather than having to use last quarter's packages...
If only this could be solved FreeBSD-wide. I would really rather not want to have to maintain a private mirror of the packages.

… I would really rather not want to have to maintain a private mirror of the packages.

▶ https://github.com/helloSystem/ISO/issues/141#issuecomment-778791853

With reference to 12.2-RELEASE-p4:

[FreeBSD-Announce] FreeBSD Errata Notice FreeBSD-EN-21:06.microcode
[FreeBSD-Announce] FreeBSD Errata Notice FreeBSD-EN-21:07.caroot
[FreeBSD-Announce] FreeBSD Errata Notice FreeBSD-EN-21:08.freebsd-update
[FreeBSD-Announce] FreeBSD Security Advisory FreeBSD-SA-21:03.pam_login_access
[FreeBSD-Announce] FreeBSD Security Advisory FreeBSD-SA-21:04.jail_remove
[FreeBSD-Announce] FreeBSD Security Advisory FreeBSD-SA-21:05.jail_chdir
[FreeBSD-Announce] FreeBSD Security Advisory FreeBSD-SA-21:06.xen

Important
FreeBSD-EN-21:08.freebsd-update fixes the bug that caused https://github.com/helloSystem/ISO/issues/115
https://github.com/helloSystem/ISO/commit/1744e6d141b91c61897f423ef815965908a85e15 prevents upgrades to packages with helloSystem and so, increases the risk of problems. Again: please revert this commit; it's excessive.

When deciding on whether to base the next version on 12.x or on 13, we need to factor in that GPU acceleration may be severly limited in 13:
https://github.com/helloSystem/ISO/issues/22#issuecomment-790815426

0.5.0 will be based on 12.2
I don't know about -p4 or how to download it.

Thanks,

I don't know about -p4

https://github.com/helloSystem/ISO/issues/135#issuecomment-781002580 above described patch levels and security advisories and:

patch level 3 i.e. 12.2-RELEASE-p3.

– so -p4 is patch level 4.
https://github.com/helloSystem/ISO/issues/135#issuecomment-784972239 above listed, and linked to, the four security vulnerabilities that were fixed at patch level 4.
I don't have a readily available list of security vulnerabilities that were fixed by levels 1, 2 or 3.
12.2-RELEASE is a huge step in the right direction ☑ however it lacks some security (through lack of all patches) and if I'm not mistaken:

https://github.com/helloSystem/ISO/commit/1744e6d141b91c61897f423ef815965908a85e15 intends to prevent use of freebsd-update(8)

If you'll prevent updates to the non-patched system, then you'll force users to be without fixes for security vulnerabilities. This is somewhat inconsistent with the welcome to the system, which mentions security:

A truly secure system involves much more than end-to-end encryption.
The prevention of package upgrades by https://github.com/helloSystem/ISO/commit/1744e6d141b91c61897f423ef815965908a85e15 is more than a security concern. It is, moreover, deeply inconsistent with the promised freedom to load software without restrictions.
Prevention is working against me, not for me.

Indeed. Will look into it.
Are you confusing freebsd-update with
% pkg update
pkg update is not yet supported in helloSystem

?

Thanks.
Briefly:

-p4 or how to download it.

I should assume that all post -p4 snapshots of STABLE are suitably patched; and more than stable enough for early development of helloSystem.
https://download.freebsd.org/ftp/snapshots/amd64/amd64/ISO-IMAGES/12.2/

and mirrors, for me the best is probably ftp://ftp.uk.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/12.2/

Patching the contents of the ISO is probably futile, because at the time when the user installs the ISO of the helloSystem release there are already new patches available. So allowing the user to patch the installed system could possibly make sense, see my thoughts in https://github.com/helloSystem/Utilities/issues/33#issuecomment-800472572 though.

https://github.com/helloSystem/ISO/issues/135#issuecomment-781002580

helloSystem documentation refers to cirrus-ci.com but it's somewhat mysterious.
How can I tell what does exist?

Found, a few weeks later:

https://cirrus-ci.com/github/helloSystem/

https://cirrus-ci.com/github/helloSystem/ shows all the builds that are going on, and their logs. Not something "mere mortals" should need to understand, but definitely interesting for helloSystem developers, testers, and power users ;-)


Missing LICENSE

I see you have no LICENSE file for this project.  The default is copyright.
I would suggest releasing the code under the GPL-3.0-or-later or AGPL-3.0-or-later license so that others are encouraged to contribute changes back to your project.

Thanks for reporting this. It is already licensed under the MIT license, which can be found at the top of most of the source files. I've just added a LICENSE file to the repository.


arch: x86: fix integer stub comments

The comment was obsolete; we simply do not allow use of the FPU or
vector math in ISRs. There is no desire to add such support, doing
this is properly offloaded to a worker thread.
Fixes #5283.
Signed-off-by: Andrew Boie<EMAIL_ADDRESS>

Codecov Report

Merging #6494 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #6494   +/-   ##
======================================
  Coverage    53.2%   53.2%
======================================
  Files         436     436
  Lines       41353   41353
  Branches     7937    7937
======================================
  Hits        22001   22001
  Misses      16128   16128
  Partials     3224    3224

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4cd0fd2...6c9e27c. Read the comment docs.


Invite user to Discord after 10 usages

I used a similar setup to the dialogue that prompts for user feedback, it looks like this:

@eyalb181 what do you think?

I'd rather not bring the "backend community" concept in at this point and keep the message focused on mirrord. How about:
"Need any help with mirrord? Come chat with our team on Discord!"


Dealing with coastlines

What's your idea for a cool feature that would help you use OHM better.
It would be great to be able to render islands and / or wetlands more reliably in OHM.
This is important b/c many coasts / shores are full of marshes before they get built up with seawalls, etc.
Perhaps I'm missing something, but it's not clear what's up.
Current workarounds
None that I can figure out:
Additional info
OHM:  https://www.openhistoricalmap.org/way/198314785#map=16/47.5644/-122.3383&layers=OD

OSM:https://www.openstreetmap.org/way/116923338#map=16/37.9331/-122.5089&layers=D

OSM: https://www.openstreetmap.org/way/318323319#map=18/37.92417/-122.50433&layers=D

This could be related to coastline processing. I'm not entirely sure but requires further inspection of the styles, vector tiles and the database.

It is relatively easy to imagine methods to solve this using endlessly nested tagging, but it would ask a lot of data editors
Specifically, one could hand land, water, land1, water1, land2, water2... And then you have to use the next iteration of whatever you are drawing on top of.
And then on the Tile/style side we’d iterate out a whole bunch to land/water20 or something and hope that It’s enough. Then add more if needed later
Pretty sure this would be derided as “tagging for the renderer” in OSM, and they’d be right.
The alternative would seem to be preprocessing water with holes for land **for every possible time slice **. Then water is on top of land always, as in other OSM maps, but we might have thousands of complex polygons in the tiles and would create a lot of other problems, like exceeding practical tile size limit and needing already to make tile sets for limited periods and swap them out.
Seems worthy of community discussion for best way forward

So.... the weird thing is that I don't think we really need to layer in this case & lemme try to explain my theory of why, although my guess is we should consult a higher power, like Seth, on this.
First is, the ocean really isn't a polygon. It's just surface that's on the wrong side of a coastline path. (Coastlines always have water to the left of their direction & generally go counterclockwise around land.
The island is an area that should be land... because it's an island. Not necessary b/c it's on a higher or lower level / layer. IOW, always draw water, unless there's an island. You can see this in the  flash of content that pops up on our maps sometimes.
The lake on the island is a relation that says, "Always draw water," no matter what's underneath.
Wait... but what about the island? Well... it's not really "on top of" the water. It's just a hole in the lake. There's nothing there.
Does this make sense?
You can drill down on OSM and see it has no special tags saying anything about landcover or that it's even land or an island. https://www.openstreetmap.org/way/290622845
If I had to put money on it, my guess is that if you said that Victoria Island (the bigger/containing island) was covered in trees, that island on the island would thus inherit those trees. Well... it wouldn't really inherit, because the island isn't really a thing. It's just a hole where the lake isn't.

Whoops... I didn't read Tim's post closely enough... it sounds like MBStudio works a little differently from the way OSM works... hmmm....

fyi - I have added some sample use cases to the ever-expanding testbed. : )

@danrademacher - who's tasked with providing the 'input needed' - is that me? Can you assign ticket w/specific asks?

We're aiming for Option 3 in above comment!

Some other solution that smarter people have to solve this problem

@geohacker is going to read up on this and talk to the person who wrote the OSM coastline pipeline

I started looking at this a bit more. The above is a bit of a hack to get coastlines to show up on top of water_areas, though like @tsinn said this may not work across the board, but I think this approach of marking coastlines as land might get us pretty close.
Using osmcoastlines is the other approach and I think that might definitely be the most accurate. We would prepare water polygons with start_date and end_date attributes and serve them in the vector tiles. These can be overlayed on top of a land background. I've reached out to Jochen who maintains the osmcoastlines to see if he has any advice for us.

Discussion with Sajjad and Jeff, focus on z9+ as shown in Leaflet, but z7-8 on vtiles:
https://vtiles.openhistoricalmap.org/#7.59/41.812/-71.148

https://vtiles.openhistoricalmap.org/#8.59/41.8117/-71.1478

Sajjad has some recommendations for how we can solve this with vector style changes only -- classifying wetlands, islands, and other "floating pieces of land" as a special layer z7+

Discussion of Sajjad, Tim, Dan today

All land and possible water polygons are in vector tiles
If we can find a smart way to handle stacking on the style, that's best.
When we tested Sajjad's Tacoma island fix, we ended up breaking pond-in-island-in-lake in the test bed.
MAYBE we could find a clever way to make separate layers  with queries into the vtiles, but needs more thinking

The weird thing is that the test bed layers water+land are working:

But the island over in Eliot Bay is not:

OTOH, with Sajjad's fix for the Eliot Bay island, the stacking in the testbed breaks.

Worth noting the that the Elliot Bay island has only natural=coastline and no place=island tag. https://openhistoricalmap.org/way/198314785#map=15/47.5598/-122.3394&layers=OD

Two tracks:

On style, we focus on having users tag all islands as place=island and then @tsinn could pull that out in our style as a separate layer on top of water areas, though we know this will solve only first order islands. But this is a major use case for coastal cities that fill in areas and make more land over time.

Meanwhile, Sajjad will talk to the Tegola team about how we can cut holes at tile creation, since others must have confronted this problem.

We have two proposal to achieve this for OHM. Both require a certain level of investment and some criteria in terms of what's possible and what's not.

Create coastline polygons on the fly in Tegola

This is probably the easiest to experiment and come out with some results. We would create a new tile layer in Tegola that queries all coastline polygons and then prepare a single multipolygon with rings for as many nesting of land/water. This would be done entirely in postgis to generate a single polygon. We will have to figure out how to use date ranges so we can produce a continuous polygon. The tile layer will then be served to the frontend. The frontend would use this as the water layer on top of land background.

Prepare OHM coastlines periodically

The long but probably most reliable approach is following OSMs footsteps and using the OSM coastline software but heavily modified for OHM. This would read a PBF and then produce multiple shapefiles for predefined date ranges. These shapefiles then will be served as vector tiles. The frontend would then apply this as a land layer.
The ohm coastlines approach requires building new infrastructure and maintenance. We would probably need to hire/consult folks from the OSM coastlines project to make sure we can achieve this.
Both approaches makes some assumptions:

We'll need to define some constraints on how date ranges (especially start of time) is defined
We may need to adjust our tag guidelines to explain how coastlines are generated
We may be heavily process polygons to make sure they are continuous and consistent so there'll be a small room for error in favour of visual cleanliness

@geohacker - thanks for this update!
My first reaction to (2) is that this won't support what most users will want, which is faster feedback on intense shoreline changes during a relatively short, punctuated timeframe (years-decades) over a fairly small area (city harbor, breakwaters, rivers draining, etc.) and that these periods will be distributed fairly evenly over post 1700 history around the world.
Does that make sense?
Then again, what sort of time interval did you have in mind for (2)? Centuries? e.g. maybe 100, 1600, 1700, 1800, 1900, 2000, etc.? And then, any changes in between shorelines is accomplished by "hacks" of water encroaching on land, etc.?
For the assumptions:

constraints - sure!
adjusting tagging - of course
processing for ensuring continuity... sounds good, but we'd need to discuss with examples, no?

@jeffreyameyer yeah 2 is not my favourite approach either and the issue you outline is true. I was thinking of a much more small intervals like ~ few months to a year (or even finer if we actually can). And yes this needs more specific examples and a proper test suite. We'll prioritise to create that before beginning any of this work.

I failed to capture here our notes from 30 Nov 2021 when we discussed another potential approach:

What if we were to say if you want to style major coastal changes around Manhattan -- make it so all the changes are inland water changes
Make the “coastline” crude and then do all the edits as “inland waterways”

And then we discussed this possible way forward:

stand up OSM coastlines based on our Planet data
Start rendering vtiles for that
No new infrastructure
Jeff will write up proposal for the community

Sajjad: Should we do a coastline cleanup campaign
For lakes-layering

those things we would still handle on the layer front, with the ordering.

If that gets too complex, then we could cut those holes on the fly

we’re a ways from rendering that. And Mapbox does a very brute force approach of layering

Ideally, PostGIS would cut the hole for you and send you the lake with the hole cut into it
Sajjad wants to ask the Tegola folks

Outcomes:

OHM community engagement on coastal cleanup
Stand up OSM coastlines for OHM
Wait on handling inland waterways layering for islands, etc

@geohacker when you are back online, take a look at these notes and see if they still seem viable as a path forward

Building on the last comment in this issue, we have a refined new proposal:

We export current natural=coastline from OHM database
Tim has defined Ice Age sea level. He uses that plus OHM coastline to make the new natural=water shapes for coastal waters. Not this could also be a simple buffer from modern coastlines, but it seems maybe a bit more logically defensible to use a baseline coastline that actually existed at one point in (pre)history

We blow away current OHM data that is natural=coastline. This is drastic and should go along with careful testing on staging and full community input.
We import Ice Age ocean as natural=coastline
We import the new coastal natural=water with maybe a default start_date of  -20000 (time of last Glacial Maximum).
Then we let the community loose on editing those new coastal edgewaters to map the history of coastal fill/change

Somewhere in 5 we probably also need to break apart the shapes in the coastal natural=coastline into smaller pieces to avoid having a single polygon that’s a ring around a whole continent or most of a continent

Splitting this out into a new issue. Underlying challenges here from the original islands concerns seem to be resolved


Add License

No license has been set for this repository.

Nevermind, it was there all along. I can't believe I missed it!


Upload issue using knora-py

While uploading data for the Portraits of Unbelonging- project with knora-py the error message below regularly shows up. The error happens when the upload is running smoothly and a lot of data already has been uploaded. When the database is cleared and the same upload requests are posted, no error is raised. I attach the trigs of the database, so the error can be reproduced. Also I'm attaching the user trigs to create the user with the permission to upload for the project. In the attached text file, there is the json that lead to the error when tried to upload.
attachments.zip
Traceback (most recent call last):
File "/Users/eric/Develop/Portraits-of-Unbelonging/src/data_import_scripts/all_file_import.py", line 519, in
c. get_data_all_file(all_file)
File "/Users/eric/Develop/Portraits-of-Unbelonging/src/data_import_scripts/all_file_import.py", line 74, in get_data_all_file
j["photograph"]=self.create_pics_from_array(pictures_in_doc, lastname)
File "/Users/eric/Develop/Portraits-of-Unbelonging/src/data_import_scripts/all_file_import.py", line 319, in create_pics_from_array
j)['iri'])
File "/Users/eric/Develop/knora-py/knora/knora.py", line 1520, in create_resource
self.on_api_error(req)
File "/Users/eric/Develop/knora-py/knora/knora.py", line 166, in on_api_error
raise KnoraError("KNORA-ERROR: status code=" + str(res.status_code) + "\nMessage:" + res.text)
knora.KnoraError: KNORA-ERROR: status code=500
Message:{"knora-api:error":"org.knora.webapi.TriplestoreResponseException: Couldn't parse Turtle from triplestore","@context":{"knora-api":"http://api.knora.org/ontology/knora-api/v2#"}}
Exception ignored in: <function Knora.del at 0x1102a5400>
Traceback (most recent call last):
File "/Users/eric/Develop/knora-py/knora/knora.py", line 156, in del
File "/Users/eric/Develop/knora-py/knora/knora.py", line 150, in logout
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py", line 158, in delete
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/api.py", line 60, in request
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 524, in request
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/sessions.py", line 700, in merge_environment_settings
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/utils.py", line 761, in get_environ_proxies
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/requests/utils.py", line 745, in should_bypass_proxies
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 2610, in proxy_bypass
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 2587, in proxy_bypass_macosx_sysconf
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/request.py", line 2534, in proxy_bypass_macosx_sysconf
ImportError: sys.meta_path is None, Python is likely shutting down

Could you also copy and paste the error from Knora's console log, showing the stack trace of the TriplestoreResponseException? We need the Scala stack trace output by the server, not the Python stack trace output by the client.

_api_1      | java.util.zip.ZipException: Not in GZIP format
api_1      | 	at java.base/java.util.zip.GZIPInputStream.readHeader(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at org.apache.http.client.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:61)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:51)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:69)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.read(Unknown Source)
api_1      | 	at java.base/java.io.InputStreamReader.read(Unknown Source)
api_1      | 	at java.base/java.io.Reader.read(Unknown Source)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.$anonfun$getSparqlHttpResponse$1(HttpTriplestoreConnector.scala:664)
api_1      | 	at scala.util.Try$.apply(Try.scala:213)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:654)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:331)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$receive$1.applyOrElse(HttpTriplestoreConnector.scala:127)
api_1      | 	at akka.actor.Actor.aroundReceive(Actor.scala:517)
api_1      | 	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.aroundReceive(HttpTriplestoreConnector.scala:64)
api_1      | 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
api_1      | 	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
api_1      | 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
api_1      | 	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
api_1      | 	at java.base/java.lang.Thread.run(Unknown Source)
api_1      | 2020-02-24 00:43:38 | ERROR | Dispatcher | Couldn't parse Turtle from triplestore
api_1      | org.knora.webapi.TriplestoreResponseException: Couldn't parse Turtle from triplestore
api_1      | 	at org.knora.webapi.TriplestoreResponseException$.apply(Exceptions.scala:315)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:338)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$receive$1.applyOrElse(HttpTriplestoreConnector.scala:127)
api_1      | 	at akka.actor.Actor.aroundReceive(Actor.scala:517)
api_1      | 	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.aroundReceive(HttpTriplestoreConnector.scala:64)
api_1      | 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
api_1      | 	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
api_1      | 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
api_1      | 	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
api_1      | 	at java.base/java.lang.Thread.run(Unknown Source)
api_1      | Caused by: org.knora.webapi.TriplestoreConnectionException: Failed to connect to triplestore
api_1      | 	at org.knora.webapi.TriplestoreConnectionException$.apply(Exceptions.scala:276)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$getSparqlHttpResponse$3.applyOrElse(HttpTriplestoreConnector.scala:690)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$getSparqlHttpResponse$3.applyOrElse(HttpTriplestoreConnector.scala:685)
api_1      | 	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
api_1      | 	at scala.util.Failure.recover(Try.scala:234)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:685)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:331)
api_1      | 	... 11 common frames omitted
api_1      | Caused by: java.util.zip.ZipException: Not in GZIP format
api_1      | 	at java.base/java.util.zip.GZIPInputStream.readHeader(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at org.apache.http.client.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:61)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:51)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:69)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.read(Unknown Source)
api_1      | 	at java.base/java.io.InputStreamReader.read(Unknown Source)
api_1      | 	at java.base/java.io.Reader.read(Unknown Source)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.$anonfun$getSparqlHttpResponse$1(HttpTriplestoreConnector.scala:664)
api_1      | 	at scala.util.Try$.apply(Try.scala:213)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:654)
api_1      | 	... 12 common frames omitted
api_1      | 2020-02-24 00:43:38 | ERROR | KnoraExceptionHandler$ | Unable to run route /v2/resources
api_1      | org.knora.webapi.TriplestoreResponseException: Couldn't parse Turtle from triplestore
api_1      | 	at org.knora.webapi.TriplestoreResponseException$.apply(Exceptions.scala:315)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:338)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$receive$1.applyOrElse(HttpTriplestoreConnector.scala:127)
api_1      | 	at akka.actor.Actor.aroundReceive(Actor.scala:517)
api_1      | 	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.aroundReceive(HttpTriplestoreConnector.scala:64)
api_1      | 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
api_1      | 	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
api_1      | 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
api_1      | 	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
api_1      | 	at java.base/java.lang.Thread.run(Unknown Source)
api_1      | Caused by: org.knora.webapi.TriplestoreConnectionException: Failed to connect to triplestore
api_1      | 	at org.knora.webapi.TriplestoreConnectionException$.apply(Exceptions.scala:276)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$getSparqlHttpResponse$3.applyOrElse(HttpTriplestoreConnector.scala:690)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$getSparqlHttpResponse$3.applyOrElse(HttpTriplestoreConnector.scala:685)
api_1      | 	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
api_1      | 	at scala.util.Failure.recover(Try.scala:234)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:685)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:331)
api_1      | 	... 11 common frames omitted
api_1      | Caused by: java.util.zip.ZipException: Not in GZIP format
api_1      | 	at java.base/java.util.zip.GZIPInputStream.readHeader(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at org.apache.http.client.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:61)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:51)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:69)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.read(Unknown Source)
api_1      | 	at java.base/java.io.InputStreamReader.read(Unknown Source)
api_1      | 	at java.base/java.io.Reader.read(Unknown Source)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.$anonfun$getSparqlHttpResponse$1(HttpTriplestoreConnector.scala:664)
api_1      | 	at scala.util.Try$.apply(Try.scala:213)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:654)
api_1      | 	... 12 common frames omitted
api_1      | 2020-02-24 00:43:38 | ERROR | OneForOneStrategy | Couldn't parse Turtle from triplestore
api_1      | org.knora.webapi.TriplestoreResponseException: Couldn't parse Turtle from triplestore
api_1      | 	at org.knora.webapi.TriplestoreResponseException$.apply(Exceptions.scala:315)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:338)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$receive$1.applyOrElse(HttpTriplestoreConnector.scala:127)
api_1      | 	at akka.actor.Actor.aroundReceive(Actor.scala:517)
api_1      | 	at akka.actor.Actor.aroundReceive$(Actor.scala:515)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.aroundReceive(HttpTriplestoreConnector.scala:64)
api_1      | 	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
api_1      | 	at akka.actor.ActorCell.invoke(ActorCell.scala:561)
api_1      | 	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
api_1      | 	at akka.dispatch.Mailbox.run(Mailbox.scala:225)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
api_1      | 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
api_1      | 	at java.base/java.lang.Thread.run(Unknown Source)
api_1      | Caused by: org.knora.webapi.TriplestoreConnectionException: Failed to connect to triplestore
api_1      | 	at org.knora.webapi.TriplestoreConnectionException$.apply(Exceptions.scala:276)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$getSparqlHttpResponse$3.applyOrElse(HttpTriplestoreConnector.scala:690)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector$$anonfun$getSparqlHttpResponse$3.applyOrElse(HttpTriplestoreConnector.scala:685)
api_1      | 	at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:38)
api_1      | 	at scala.util.Failure.recover(Try.scala:234)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:685)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.org$knora$webapi$store$triplestore$http$HttpTriplestoreConnector$$sparqlHttpExtendedConstruct(HttpTriplestoreConnector.scala:331)
api_1      | 	... 11 common frames omitted
api_1      | Caused by: java.util.zip.ZipException: Not in GZIP format
api_1      | 	at java.base/java.util.zip.GZIPInputStream.readHeader(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at java.base/java.util.zip.GZIPInputStream.(Unknown Source)
api_1      | 	at org.apache.http.client.entity.GZIPInputStreamFactory.create(GZIPInputStreamFactory.java:61)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.initWrapper(LazyDecompressingInputStream.java:51)
api_1      | 	at org.apache.http.client.entity.LazyDecompressingInputStream.read(LazyDecompressingInputStream.java:69)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.readBytes(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.implRead(Unknown Source)
api_1      | 	at java.base/sun.nio.cs.StreamDecoder.read(Unknown Source)
api_1      | 	at java.base/java.io.InputStreamReader.read(Unknown Source)
Is this what you are looking for?
api_1      | 	at java.base/java.io.Reader.read(Unknown Source)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:227)
api_1      | 	at org.apache.http.util.EntityUtils.toString(EntityUtils.java:308)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.$anonfun$getSparqlHttpResponse$1(HttpTriplestoreConnector.scala:664)
api_1      | 	at scala.util.Try$.apply(Try.scala:213)
api_1      | 	at org.knora.webapi.store.triplestore.http.HttpTriplestoreConnector.getSparqlHttpResponse(HttpTriplestoreConnector.scala:654)
api_1      | 	... 12 common frames omitted_

Yes, that's it. Very strange. It looks as if the triplestore returned a Content-Encoding: gzip header, but without returning a response in GZIP format.

Could that be a mistake on my side? Or is this an issue with knora-api?

Von: Benjamin Geer<EMAIL_ADDRESS>Gesendet: Dienstag, 25. Februar 2020 12:08:11
An: dasch-swiss/knora-api
Cc: Eric Sommerhalder; Author
Betreff: Re: [dasch-swiss/knora-api] Upload memory issue using knora-py (#1605)
Yes, that's it. Very strange. It looks as if the triplestore returned a Content-Encoding: gzip header, but without returning a response in GZIP format.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/dasch-swiss/knora-api/issues/1605?email_source=notifications&email_token=AJFZAPBOPHTLBZYOWNKI42LRET3ZXA5CNFSM4K3EJRH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEM3RSKY#issuecomment-590813483, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AJFZAPCH2YWS7AE3APFN2RDRET3ZXANCNFSM4K3EJRHQ.

My guess would be that it's a GraphDB bug, but I'll try to reproduce it and find out what's going wrong.
The JSON file you uploaded is actually an RTF file and looks like this:
{\rtf1\ansi\ansicpg1252\cocoartf1671\cocoasubrtf400
{\fonttbl\f0\fswiss\fcharset0 Helvetica;}
{\colortbl;\red255\green255\blue255;}
{\*\expandedcolortbl;;}
\paperw11900\paperh16840\margl1440\margr1440\vieww10800\viewh8400\viewkind0
\pard\tx566\tx1133\tx1700\tx2267\tx2834\tx3401\tx3968\tx4535\tx5102\tx5669\tx6236\tx6803\pardirnatural\partightenfactor0

Please upload it again as a plain text file (you can use the extension .json).
Also please give more specific instructions for what to do with these files to reproduce the error, e.g. (1) Run script XYZ with these arguments, (2) run curl with these arguments...

This error is getting very weird. Even though I'm uploading the exact same json into a database with the exact same contents, no error is raised. Here what I'm doing:

Start knora-api with make stack-up
Go to the GraphDB interface: http://<IP_ADDRESS>:7200
Go to import -> rdf and import the three trigs in the order of statements-3.trig, pou-admin-data.trig,  pou-permission-data.trig
Restart knora-api with make stack-up
Run the attached python script, that uploads the problematic JSON that raised the error. Note that the scripts needs you to have knora.py installed

reproduce_error.py.zip
However, this doesn't raise the error.

Thanks Eric, OK, so we have an intermittent problem. That suggests that maybe it really is related to something like memory use, which is not constant. We can try to reproduce it by running the same test multiple times to see if we can make the error occur after a while. Could you put a loop in reproduce_error.py and see what happens if it tries 100 times?

Did so, 100 went through trying with 10000.
I'm attaching the original python script I used for the upload. The error is usually raised in line 303. The arrays in that area of the code can get quiet large. The error occured only in cases where these arrays where quiet large and a lot of create_resource-requests where posted.
Could this be a reason for the error?
all_file_import.py.zip

Could this be a reason for the error?

I don't think this error has anything to do with knora-py or your Python code. It involves the communication between Knora and the triplestore: Knora makes an HTTP request to the triplestore, and the triplestore returns an unexpected response. I think we will have to send a bug report to GraphDB, but it's hard to do that if we can't reproduce the error.

Ok so the error occurs with a loop after approximately
1530 iterations. However when I do another iteration after the program has failed, no error is raised. I have attached the python script with the loop.
reproduce_error.py.zip

Restart knora-api with make stack-up

To restart knora after loading data, please use make stack-restart-api.

just a few thoughts when getting these random errors on import, I first check:

graphdb's repo settings (in the workbench) Query time-out and Limit query results
knora to graphdb connection pool size, KNORA_WEBAPI_DB_CONNECTIONS

For small configs and large dataset, as a rule of thumb I would put Query time-out to 60s, Limit query results to 5000000 and KNORA_WEBAPI_DB_CONNECTIONS to 2.

@loicjaouen OK, but still, if there's a timeout somewhere, it shouldn't result in the error Not in GZIP format.

I'm unable to upload statements-3.trig: the upload never finishes, and a lot of error messages appear in the GraphDB workbench:

Normally with a large file I would use the "Server Files" option, but I don't see how to do this if I'm running GraphDB in Docker. @subotic any ideas?

@subotic Since I can't import the test data into GraphDB in Docker (see above), I wanted to try this with GraphDB running outside of Docker, but for that I would need to be able to run Sipi, and and I can't because of #1614.

@benjamingeer

in the workbench, find the folder where the graphdb expect the (import -> RDF -> server files -> help)
copy the files using docker cp

@loicjaouen Thanks. Ivan told me that there's an environment variable, KNORA_GDB_IMPORT, described in the Knora README, that I can set so GraphDB in Docker will import files from the specified directory.

@subotic Using KNORA_GDB_IMPORT, I still can't import statements-3.trig. GraphDB seems to crash. The log says:
db_1       | The license key permits the use of GraphDB on systems with up to 1 CPU cores, but this Java Virtual Machine reports that 4 CPU cores are available.
db_1       | [INFO ] 2020-03-03 14:11:34,026 [import-task-knora-test-1 | c.o.r.p.ParallelRDFInserter] Importing stream
db_1       | [WARN ] 2020-03-03 14:11:34,044 [import-task-knora-test-1 | c.o.r.p.ParallelLoader] We were going to use 4 threads for inference but your license allows only 1, throttling applied
db_1       | [INFO ] 2020-03-03 14:11:44,029 [import-task-knora-test-1 | c.o.r.p.ParallelRDFInserter] Parsed 322,294 stmts. Rate: 32,226 st/s. Statements overall: 322,294. Global average rate: 32,222 st/s. Now: Tue Mar 03 14:11:44 UTC 2020. Total memory: 4929M, Free memory: 4064M, Max memory: 4929M.
docker_db_1 exited with code 137

Could you possibly try it and see if you get the same error?

I just ran into the same error again. I was able to fix it by simply running make stack-down followed by make stack-up

With GraphDB running outside of Docker, I can import the TriG files, but when I run all_file_import.py, I get this error:
FileNotFoundError: [Errno 2] No such file or directory: 'current_data/All renunciation of nationality cases_only_photographs_attached_11_02.csv'

Sorry, for that, of course you need the file. I adjusted the script a bit so you don't need other files as well. Please find attached the script as well as the file.
all_file_simplified.zip

Thanks, but now I get:
FileNotFoundError: [Errno 2] No such file or directory: 'hicri_conversion_table.csv'

Ok, tried on my machine now, this should work now:
all_file_simplified.zip

Closing because this seems to be a GraphDB issue, and we're not using GraphDB anymore.


Persist

Added persistence convenience functions (Save, Load, and SaveFile/LoadFile).
The wallet is next since it's more complicated. It will use a database rather than a flat JSON file.

I don't know that 'files.go' is the best name. On Unix, everything is a file. Perhaps 'humanreadable.go' or something in that spirit. Or even 'json.go'.

agreed, I couldn't really think of a name to capture what those functions are really doing, which is encoding data and prepending it with a standard header.


CkCallback::ckExit does not work with interop mode

Original issue: https://charm.cs.illinois.edu/redmine/issues/2063

Instead of calling CkExit, which is redefined by mpi-interoperate.h to only exit the library and stop the charm scheduler, it calls the full CkExit and exits the program.

Original date: 2019-04-12 17:19:26

This is probably a matter of undefining (and redefining?) the macro magic to allow an optional parameter in C.

Original date: 2019-04-19 03:54:32

Messing with the macro wouldn't really work. It's the fact that the CkExit macro is redefined for interop mode. But that redef is only visible to files which include the interop header. CkCallback does not, so it calls regular CkExit instead of LibCkExit. I don't think this change is feasible to do at compile time, but considering its a one time callback, I don't think that matters.
Implementation here: ~~https://charm.cs.illinois.edu/gerrit/c/charm/+/5106~~ https://github.com/UIUC-PPL/charm/commit/35cdcdd045086dc13e3ce2e6054a77c4f97c788f


Webgl Support

Does it support webgl?

At the moment, it's quite difficult because Barracuda on WebGL only supports the naive CSharp backend (very slow). I'd like to reevaluate it when Burst gets available on WebGL.


With tightly packed macros, LVS fails due to hanging strips of metal on the PDN

Description
With tightly packed macros, LVS fails due to hanging strips of metal on the PDN

Environment
Kernel: Linux v5.13.0-40-generic
Distribution: ubuntu 20.04
Python: v3.8.10 (OK)
Container Engine: docker v20.10.7 (OK)
OpenLane Git Version: -dev
pip:click: INSTALLED
pip:pyyaml: INSTALLED
pip:venv: INSTALLED
---
PDK Version Verification Status: OK
---
Git Log (Last 3 Commits)

e4bfdd7 2022-02-22T16:59:14-03:00 Remove pip install, no longer needed (#953) - Vitor Bandeira -  (grafted, HEAD, tag: 2022.02.23_02.50.41)

Reproduction Material

tarball not easy as this is Caravel - see repo: https://github.com/mattvenn/tapeout_100
List the commands used to run the design:
make install  (installs caravel submodule)
make uncompress (uncompresses gds files)
user_project_wrapper

Expected behavior
LVS shouldn't fail
Logs
final lines of user_project_wrapper/runs/user_project_wrapper/logs/finishing/26-user_project_wrapper.lef.log:
vssd1                                      |(no matching pin)
vssd1_uq0                                  |(no matching pin)
vssd1_uq1                                  |(no matching pin)
vssd1_uq2                                  |(no matching pin)
vssd1_uq3                                  |(no matching pin)

@mattvenn
I am trying to clone your repo with https://github.com/The-OpenROAD-Project/OpenLane-MPW-CI and running its got failed.
[INFO]: Preparing LEF files for the max corner...
[INFO]: Looking for files defined in ::env(EXTRA_GDS_FILES)  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_function_generator.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_vga_clock.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_frequency_counter.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_rgb_mixer.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_hack_soc_dffram.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_teras.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_alu74181.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_vgademo_on_fpga.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_silife.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_acorn_prng.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_hsv_mixer.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_skullfet.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wb_bridge_2way.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wb_openram_wrapper.gds  /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/sky130_sram_1kbyte_1rw1r_32x256_8.gds  ...
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_function_generator.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_vga_clock.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_frequency_counter.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_rgb_mixer.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_hack_soc_dffram.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_teras.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_alu74181.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_vgademo_on_fpga.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_silife.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_acorn_prng.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_hsv_mixer.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wrapped_skullfet.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wb_bridge_2way.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/wb_openram_wrapper.gds exists.
[INFO]: /openlane/designs/tapeout_100/openlane/user_project_wrapper/../../gds/sky130_sram_1kbyte_1rw1r_32x256_8.gds exists.
[STEP 1]
[INFO]: Running Synthesis...
[STEP 2]
[INFO]: Running Single-Corner Static Timing Analysis...
[INFO]: Creating a netlist with power/ground pins.
[STEP 3]
[INFO]: Running Initial Floorplanning...
[ERROR]: Floorplanning failed
[ERROR]: module wrapped_icestudio_test not found in /openlane/designs/tapeout_100/runs/RUN_2022.04.30_09.48.18/tmp/merged.nom.lef
[ERROR]: Check whether EXTRA_LEFS is set appropriately
[INFO]: Generating final set of reports...
[INFO]: Created manufacturability report at 'runs/RUN_2022.04.30_09.48.18/reports/manufacturability.rpt'.
[INFO]: Created metrics report at 'runs/RUN_2022.04.30_09.48.18/reports/metrics.csv'.
[INFO]: Saving runtime environment...
[ERROR]: Flow failed.

    while executing
"flow_fail"
    (procedure "check_floorplan_missing_lef" line 11)
    invoked from within
"check_floorplan_missing_lef"
    (procedure "init_floorplan" line 12)
    invoked from within
"init_floorplan"
    (procedure "run_floorplan" line 7)
    invoked from within
"[lindex $step_exe 0] [lindex $step_exe 1] "
    (procedure "run_non_interactive_mode" line 55)
    invoked from within
"run_non_interactive_mode {*}$argv"
    invoked from within
"if { [info exists flags_map(-interactive)] || [info exists flags_map(-it)] } {
	if { [info exists arg_values(-file)] } {
		run_file [file normalize $a..."
    (file "./flow.tcl" line 427)
make: *** [quick_run] Error 1
make: Leaving directory `/home/vijayan/MPW_CI/OpenLane-MPW-CI/OpenLane'

Can you update required openlane/user_project_wrapper/config.tcl to run the flow to re-produce your issue at our end?

Why does it being caravel make it hard to produce a tarball?  @donn any thoughts?

You can ask the caravel team.

@maliberty because the caravel_user_project is 8.2G. So I need to know what to .tar.gz in order to capture what you need for a test case

@vijayank88 thanks for taking a look at this. I've fixed the example repo, my bad I should have tested it locally.

I'm not sure it's the best way, but I'll throw an exit 1 into the step that I want to debug, and Openlane should die and populate  openroad_issue_reproducible. Assuming things start going wrong at the PDN stage, add it to scripts/openlane/pdn.tcl.
@donn or @maliberty might have a better idea.

actually I did know this trick but thought it would only work with openlane, rather than inside caravel_user_project. Thanks Anton!

@mattvenn
I cloned your repo and its fail with  LVS mismach error.
Its similar to #1067

I'm not sure it's the best way, but I'll throw an exit 1 into the step that I want to debug, and Openlane should die and populate openroad_issue_reproducible. Assuming things start going wrong at the PDN stage, add it to scripts/openlane/pdn.tcl.
@donn or @maliberty might have a better idea.

This way its works and produce openroad_issue_reproducible, but user should remove exit 1 from both script.

@maliberty
Fixing #1067  will resolve this error too.

I don't think so. I think this is valid that netgen is showing an error as there are many unconnected strips of metal. The problem is that the PDN is generating these hanging strips that don't connect to anything.

Can you try this with https://github.com/The-OpenROAD-Project/OpenLane/pull/1059 ?

@antonblanchard
Here @mattvenn  re-used macros wrapped_icestudio_test 7 times.
How to define MACRO_HOOKS?
set ::env(FP_PDN_MACRO_HOOKS) "\
	mprj vccd1 vssd1"

@vijayank88 @mattvenn I think you want:
set ::env(FP_PDN_MACRO_HOOKS) "\
	wrapped_icestudio_test_0 vccd1 vssd1, \
	wrapped_icestudio_test_1 vccd1 vssd1, \
	wrapped_icestudio_test_2 vccd1 vssd1, \
	wrapped_icestudio_test_3 vccd1 vssd1, \
	wrapped_icestudio_test_4 vccd1 vssd1, \
	wrapped_icestudio_test_5 vccd1 vssd1, \
	wrapped_icestudio_test_6 vccd1 vssd1, \
	wrapped_icestudio_test_7 vccd1 vssd1"

@maliberty I notice the Openlane documentation claims macros are connected to the first power domain if they are unspecified:
| `FP_PDN_MACRO_HOOKS` | Specifies explicit power connections of internal macros to the top level power grid. Comma separated list of macro instance names and power domain vdd and ground net names: `<instance_name> <vdd_net> <gnd_net>`
 (Default: macros are connected to the first power domain

That's not correct is it? Should we remove that comment?

@antonblanchard re "Should we remove that comment?" Yes.  I'm not clear that was actually true before either but I haven't tested it.  Randomly connecting macros seems dangerous.

@antonblanchard re "Should we remove that comment?" Yes. I'm not clear that was actually true before either but I haven't tested it. Arbitrarily connecting macros seems dangerous.

Created https://github.com/The-OpenROAD-Project/OpenLane/pull/1091

@maliberty Latest PDN stage attached here:
openroad_issue_reproducible.zip

@arlpetergadfort there are dangling pieces of met4 between the macros:

The right piece has a via but the left does not.  The right belongs to the single met5 vdd strap while the left is part of vss and has nothing to connect to.

@vijayank88 @mattvenn I think you want:
set ::env(FP_PDN_MACRO_HOOKS) "\
	wrapped_icestudio_test_0 vccd1 vssd1, \
	wrapped_icestudio_test_1 vccd1 vssd1, \
	wrapped_icestudio_test_2 vccd1 vssd1, \
	wrapped_icestudio_test_3 vccd1 vssd1, \
	wrapped_icestudio_test_4 vccd1 vssd1, \
	wrapped_icestudio_test_5 vccd1 vssd1, \
	wrapped_icestudio_test_6 vccd1 vssd1, \
	wrapped_icestudio_test_7 vccd1 vssd1"

@maliberty I notice the Openlane documentation claims macros are connected to the first power domain if they are unspecified:
| `FP_PDN_MACRO_HOOKS` | Specifies explicit power connections of internal macros to the top level power grid. Comma separated list of macro instance names and power domain vdd and ground net names: `<instance_name> <vdd_net> <gnd_net>`
 (Default: macros are connected to the first power domain

That's not correct is it? Should we remove that comment?

I've found through experimentation that the FP_PDN_MACRO_HOOKS isn't required - macros get connected to power anyway. What does it do? I've also tried using it, and I still get warnings about macros not being powered.

@vijayank88 I tried your suggestions and got this new error:
[ERROR PDN-0165] Conflict found, instance wrapped_icestudio_test_0 is part of two grid definitions (CORE_macro_grid_2, CORE_macro_grid_1)

@mattvenn Hey matt, did you maybe manage to fix this issue? (Conflict found...) I am getting the same issue.

No, haven't found a fix yet.

@mattvenn
Its look like new issue. Can you please create new issue with test case?

@mattvenn
We need your test case. @jurevreca12 issue got fixed.

@vijayank88 I noticed one thing after closing the issue. I commented out the macro set ::env(FP_PDN_MACRO_HOOKS) . If I don't do that i  still get the same error as Matt. Is this a problem?

@jurevreca12
Open a new issue with clear description and test case.

https://github.com/The-OpenROAD-Project/OpenLane/issues/1104

this issue has the 'waiting on OP' tag applied. But I'm still stuck on this issue as noted above on May 23.
With the config changes suggested by @vijayank88  I get the new error:
Conflict found, instance wrapped_icestudio_test_0 is part of two grid definitions (CORE_macro_grid_2, CORE_macro_grid_1).

I just checked @jurevreca12 's comment. By commenting out the FP_PDN_MACRO_HOOKS but leaving the LVS_CONNECT_BY_LABEL the flow completes.

@mattvenn LVS_CONNECT_BY_LABEL means that unconnected nets having the same label are treated as connected.
This is not something I'd recommend on a top level block. For example, even if your macros all had vccd1 labels on the power supplies, you still want them to be physically connected. With this option, even if they are not connected LVS, will pass.

Thank you Mitch. Yes let's leave this open then as the LVS_CONNECT_BY_LABEL
is definitely something I'll only use for prototyping
On Fri, 24 Jun 2022 at 17:54, Mitch Bailey @.***> wrote:

@mattvenn https://github.com/mattvenn LVS_CONNECT_BY_LABEL means that
unconnected nets having the same label are treated as connected.
This is not something I'd recommend on a top level block. For example,
even if your macros all had vccd1 labels on the power supplies, you still
want them to be physically connected. With this option, even if they are
not connected LVS, will pass.
—
Reply to this email directly, view it on GitHub
https://github.com/The-OpenROAD-Project/OpenLane/issues/1076#issuecomment-1165707623,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAE223B4QVASNC6CANLVNNLVQXK2VANCNFSM5UWP4YLA
.
You are receiving this because you were mentioned.Message ID:
@.***>

--
Matthew Venn
web mattvenn.net twitter @matthewvenn https://twitter.com/matthewvenn

@mattvenn This is an old open issue. I see it was referenced in multiple other issues and PRs and might not be relevant anymore. Is there anything to look for here ?

This is too out of date now. If there is still an bug please open a new issue


v8: fix 4.5.107 build

Motivation for this change
Fix v8-4.5.107 build
Things done

[ ] Tested using sandboxing
(nix.useChroot on NixOS,
or option build-use-chroot in nix.conf
on non-NixOS)
Built on platform(s)

[ ] NixOS
[ ] OS X
[x] Linux

[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
[x] Tested execution of all binary files (usually in ./result/bin/)
[x] Fits CONTRIBUTING.md.

Similar to #14272, but fixes 4.5 build rather than generic.

Ignores errors due to strict-overflow warnings
Strips clang-only '-Wno-format-pedantic' flag out since this build
uses gcc

 Built locally just fine, where current master does not. I'd say merge it, but I'll wait until Monday afternoon to give s someone who knows a bit more about it to look.

Reviewed 1 of 1 files at r1.
Review status: all files reviewed at latest revision, all discussions resolved.

Comments from Reviewable


merge_dense_input  in input_embedding.py

Describe the question(问题描述)
It's difficult for me to understand doing embedding for a dense feature,Why we need do that?thx !
Additional context
Add any other context about the problem here.
Operating environment(运行环境):

python version [e.g. 3.6]
tensorflow version [e.g. 1.4.0,]
deepctr version [e.g. 0.3.2,]

hi @Phoenixinfire please refer to
https://deepctr-doc.readthedocs.io/en/latest/Quick-Start.html#step-3-generate-feature-config-dict
Whether doing embedding for dense feature or concat dense feautre with sparse embedding vectors
depends on data and tasks.

ok ,thank you very much.


Test whether Minio keys are being set correctly

Saw this error when I passed in the default creds; want to test to see if we still get that error when we choose non-default creds

Tried that and now got this error???

https://stackoverflow.com/questions/61343885/minio-unable-to-initialize-server-switching-into-safe-mode-unable-to-initialize
... not super helpful

@mattstauffer are you still having issues with it? I just tried to spin up a Minio instance and it's working over here... I do get some warnings in the logs, but those don't seem to be actual issues (as in things we need to fix)


⚠ Jenkins Plugins Artifactory has degraded performance

In b8e1431, Jenkins Plugins Artifactory (https://get.jenkins.io/plugins/artifactory/) experienced degraded performance:

HTTP code: 200
Response time: 1201 ms

Resolved: Jenkins Plugins Artifactory performance has improved in d26538a after 23 minutes.


Hotfix 2.0.2

Not to merge, just to test; for closing #794.

Running tests by hand because github. (https://github.community/t/run-actions-on-pull-requests-with-merge-conflicts/17104/2).


Fix _WIN32 definition for Python

Windows code have typo that causes build to fail:
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-_WIN32'

When passing defines from command line they must be prepended with -D but it was prepended only by -.

Updated description, hope it's clear now.


ci: fix broken microsoft repo

temporary fix broken microsoft repo
https://github.com/microsoft/linux-package-repositories/issues/130

what microsoft repo?

what microsoft repo?

An Ubuntu mirror used by the github runners I guess, could we just wait out? Sounds pretty bad, would  be surprised if they fix longer than any routine github outage anyway.

what microsoft repo?

Apparently GitHub runners have these Microsoft APT repositories added by default.

could we just wait out?

Given that there are already over 20 PR runs affected by this and this will likely keep piling up, we would want a temporary fix in ASAP.


Release/0.13.0

Closes #84
Release candidate is available for verification
https://pypi.org/project/python-cmr/0.13.0rc1/
https://github.com/nasa/python_cmr/releases/tag/v0.13.0-rc1

Yeah I auto-formatted using PyCharm to help clear some linting errors. Seems like it over-indented in some places

No worries. I thought they came from one of the PRs in the bunch. Do we want to create an issue for updating our process to use ruff? (We might need to do some reformatting to deal with long lines and perhaps a few other things that we currently let flake8 flag without failing the build.)

I haven't used ruff yet but yeah I'm not tied to flake8 as a linter... as long as we're running a linter I'm happy


Added agat/spflagshortintrons and updated modules

PR checklist
Closes #XXX

[x] Added agat/spflagshortintrons and updated modules
[x] Followed the module conventions in the contribution docs
[x] Use BioConda and BioContainers if possible to fulfil software requirements.

For visibility, https://github.com/nf-core/modules/issues/6963